Oct 29 21:45:54.307963 ip-10-0-136-38 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 21:45:54.693360 ip-10-0-136-38 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:45:54.693360 ip-10-0-136-38 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 21:45:54.693360 ip-10-0-136-38 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:45:54.693360 ip-10-0-136-38 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:45:54.698160 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698038    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:54.698160 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698153    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:54.698160 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698158    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:54.698160 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698161    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:54.698160 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698166    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:54.698160 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698170    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:54.698160 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698173    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698179    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698185    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698189    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698192    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698195    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698199    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698202    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698207    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698210    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698214    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698218    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698221    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698224    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698236    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698240    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698243    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698247    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698250    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:54.698616 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698254    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698257    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698261    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698264    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698268    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698271    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698277    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698281    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698285    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698289    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698293    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698297    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698303    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698307    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698312    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698317    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698320    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698324    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698328    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:54.699344 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698332    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698336    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698340    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698347    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698354    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698359    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698364    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698369    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698373    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698377    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698381    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698386    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698390    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698394    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698398    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698406    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698410    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698415    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698419    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698423    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:54.700131 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698427    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698432    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698436    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698441    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698446    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698450    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698456    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698461    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698465    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698469    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698474    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698479    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698483    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698487    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698491    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698496    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698500    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698504    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698509    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698513    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:54.700956 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698517    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698521    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698525    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698532    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698539    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698564    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698569    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698574    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698578    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698583    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698588    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698592    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698596    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698601    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698605    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698609    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698613    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698617    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698621    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:54.701438 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698629    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698634    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.698638    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700075    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700091    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700100    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700105    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700110    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700115    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700120    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700124    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700128    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700132    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700136    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700140    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700144    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700148    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700152    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700157    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:54.701925 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700161    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700165    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700169    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700174    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700178    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700182    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700186    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700191    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700195    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700199    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700203    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700207    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700212    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700217    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700222    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700227    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700231    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700235    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700240    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700244    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:54.702413 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700248    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700253    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700257    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700261    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700265    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700270    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700274    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700278    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700282    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700286    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700290    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700295    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700299    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700303    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700307    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700311    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700316    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700320    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700324    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700328    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:54.702924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700332    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700337    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700341    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700345    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700349    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700354    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700358    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700363    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700368    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700372    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700376    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700381    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700385    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700389    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700393    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700398    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700402    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700406    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700410    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:54.703444 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700415    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700419    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700423    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700429    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700433    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700437    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700441    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700445    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700452    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700456    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700460    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700464    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700468    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700472    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700477    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700484    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700490    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700494    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700499    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700504    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:54.703935 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700509    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700514    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700518    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700523    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700529    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700533    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700536    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700558    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700564    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700568    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.700572    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701357    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701373    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701376    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701380    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701383    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701386    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701389    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701391    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701394    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:54.704461 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701396    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701399    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701402    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701404    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701407    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701409    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701412    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701415    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701417    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701420    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701422    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701425    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701427    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701430    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701433    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701435    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701438    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701441    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701443    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:54.704966 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701446    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701449    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701451    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701454    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701457    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701462    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701464    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701467    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701469    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701472    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701476    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701481    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701484    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701487    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701490    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701492    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701495    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701497    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701500    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:54.705410 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701502    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701505    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701507    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701510    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701513    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701515    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701518    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701520    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701523    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701525    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701528    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701531    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701533    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701536    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701538    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701541    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701557    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701561    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701564    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701568    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:54.705892 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701571    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701574    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701576    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701579    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701582    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701586    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701589    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701591    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701594    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701597    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701599    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701602    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701605    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701607    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701609    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701612    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701615    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701617    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701620    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701622    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:54.706368 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701625    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701628    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701630    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701632    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701635    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701638    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701640    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701643    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701647    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701652    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701655    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701658    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701663    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701666    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701669    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701671    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701674    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701677    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.701679    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702787    2582 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 21:45:54.706895 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702809    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702817    2582 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702822    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702828    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702832    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702837    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702841    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702845    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702848    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702852    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702856    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702859    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702863    2582 flags.go:64] FLAG: --cgroup-root=""
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702870    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702873    2582 flags.go:64] FLAG: --client-ca-file=""
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702876    2582 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702880    2582 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702885    2582 flags.go:64] FLAG: --cluster-domain=""
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702889    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702892    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702896    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702901    2582 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702906    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 21:45:54.707413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702910    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702913    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702918    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702922    2582 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702925    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702928    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702932    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702935    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702940    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702944    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702947    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702950    2582 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702953    2582 flags.go:64] FLAG: --enable-server="true"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702956    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702961    2582 flags.go:64] FLAG: --event-burst="100"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702964    2582 flags.go:64] FLAG: --event-qps="50"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702968    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702971    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702974    2582 flags.go:64] FLAG: --eviction-hard=""
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702979    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702982    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702985    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702989    2582 flags.go:64] FLAG: --eviction-soft=""
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702993    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702996    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 21:45:54.708035 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.702999    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703002    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703005    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703008    2582 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703011    2582 flags.go:64] FLAG: --feature-gates=""
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703016    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703019    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703023    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703026    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703030    2582 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703033    2582 flags.go:64] FLAG: --help="false"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703037    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-136-38.ec2.internal"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703040    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703043    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703047    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703050    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703054    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703057    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703060    2582 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703063    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703065    2582 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703068    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703071    2582 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703074    2582 flags.go:64] FLAG: --kube-reserved=""
Oct 29 21:45:54.708707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703077    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703080    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703083    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703086    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703089    2582 flags.go:64] FLAG: --lock-file=""
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703097    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703100    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703104    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703110    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703113    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703116    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703119    2582 flags.go:64] FLAG: --logging-format="text"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703124    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703127    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703131    2582 flags.go:64] FLAG: --manifest-url=""
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703134    2582 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703139    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703142    2582 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703146    2582 flags.go:64] FLAG: --max-pods="110"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703150    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703153    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703157    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703160    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703162    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703166    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 21:45:54.709290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703169    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703179    2582 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703182    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703185    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703188    2582 flags.go:64] FLAG: --pod-cidr=""
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703191    2582 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703194    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703196    2582 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703199    2582 flags.go:64] FLAG: --port="10250"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703202    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703205    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b2927172f04d03f2"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703208    2582 flags.go:64] FLAG: --qos-reserved=""
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703213    2582 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703217    2582 flags.go:64] FLAG: --register-node="true"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703220    2582 flags.go:64] FLAG: --register-with-taints=""
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703224    2582 flags.go:64] FLAG: --registry-burst="10"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703227    2582 flags.go:64] FLAG: --registry-qps="5"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703231    2582 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703235    2582 flags.go:64] FLAG: --reserved-memory=""
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703239    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703242    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703245    2582 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703248    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703251    2582 flags.go:64] FLAG: --runonce="false"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703254    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703257    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703260    2582 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 21:45:54.709918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703264    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703267    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703270    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703274    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703277    2582 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703280    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703283    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703286    2582 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703289    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703292    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703295    2582 flags.go:64] FLAG: --system-cgroups=""
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703298    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703305    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703308    2582 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703311    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703316    2582 flags.go:64] FLAG: --tls-min-version=""
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703319    2582 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703326    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703330    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703333    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703336    2582 flags.go:64] FLAG: --v="2"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703341    2582 flags.go:64] FLAG: --version="false"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703354    2582 flags.go:64] FLAG: --vmodule=""
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703359    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703362    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 21:45:54.710647 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703521    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703526    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703530    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703532    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703535    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703538    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703556    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703560    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703564    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703567    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703570    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703572    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703575    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703577    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703580    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703582    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703585    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703588    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703590    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703597    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:54.711256 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703600    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703602    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703605    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703607    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703612    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703614    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703617    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703620    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703622    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703625    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703628    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703630    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703633    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703636    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703638    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703641    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703646    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703649    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703652    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:54.711801 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703655    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703657    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703661    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703664    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703666    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703669    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703671    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703674    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703676    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703679    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703682    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703684    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703689    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703691    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703695    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703697    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703700    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703705    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703707    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:54.712535 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703710    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703713    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703715    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703718    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703720    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703723    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703725    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703728    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703731    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703733    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703736    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703738    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703741    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703744    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703746    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703749    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703752    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703755    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703757    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:54.713314 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703760    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703762    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703765    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703767    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703770    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703772    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703775    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703778    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703781    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703783    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703785    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703790    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703792    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703795    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703797    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703800    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703803    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703805    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703808    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:54.713905 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703811    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703814    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703816    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703819    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703822    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703824    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703827    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703830    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703832    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703835    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.703870    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703911    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703916    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703919    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703922    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703925    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:54.714390 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703927    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703930    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703932    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703935    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703939    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703942    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703945    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703948    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703952    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703954    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703957    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703960    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703963    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703965    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703968    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703970    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703973    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703976    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703978    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:54.714794 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703981    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703983    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703986    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.703989    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704002    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704006    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704009    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704011    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704014    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704017    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704020    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704024    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704028    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704031    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704034    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704037    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704040    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704044    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704047    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704049    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:54.715238 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704051    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704055    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704058    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704062    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704065    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704067    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704070    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704073    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704079    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704082    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704084    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704087    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704089    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704092    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704095    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704097    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704100    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704102    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704105    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704108    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:54.715730 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704111    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704113    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704118    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704121    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704123    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704126    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704129    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704131    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704134    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704139    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704141    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704144    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704146    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704150    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704153    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704155    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704158    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704161    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704164    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704166    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:54.716222 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704169    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704172    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704175    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704177    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704180    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704182    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704184    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704187    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704189    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704192    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704195    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704197    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704199    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704203    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704206    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704209    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704211    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704214    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704216    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:54.716725 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704219    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704221    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.704226    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.714259    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.714283    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.714993    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.715001    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715076    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715083    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715092    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715096    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715099    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715102    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715104    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715107    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715110    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715113    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715115    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715120    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715125    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:54.717163 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715128    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715132    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715135    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715138    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715141    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715144    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715146    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715149    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715152    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715155    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715157    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715161    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715165    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715168    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715173    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715176    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715179    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715181    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715185    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:54.717763 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715187    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715191    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715194    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715196    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715199    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715203    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715205    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715208    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715210    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715213    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715216    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715218    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715221    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715223    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715228    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715230    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715233    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715236    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715238    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715241    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:54.718203 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715244    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715246    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715250    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715252    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715255    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715258    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715260    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715263    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715266    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715268    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715271    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715275    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715278    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715281    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715283    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715286    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715288    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715291    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715293    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715296    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:54.718699 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715298    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715300    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715303    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715306    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715308    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715311    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715313    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715317    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715319    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715322    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715324    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715326    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715329    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715332    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715335    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715337    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715340    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715342    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715346    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:54.719182 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715348    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715351    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715354    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715357    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715360    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715362    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715365    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715368    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715370    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715373    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715375    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715378    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715380    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715383    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715385    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.715433    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:45:54.719713 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715617    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715625    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715629    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715632    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715635    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715638    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715641    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715644    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715647    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715649    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715652    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715654    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715657    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715659    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715663    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715665    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715669    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715671    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715674    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:54.720152 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715676    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715681    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715686    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715690    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715692    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715695    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715698    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715700    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715703    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715706    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715708    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715713    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715716    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715718    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715721    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715723    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715726    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715729    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715731    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:54.720635 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715736    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715739    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715742    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715744    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715747    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715750    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715752    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715755    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715757    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715760    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715763    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715766    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715768    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715772    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715775    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715777    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715780    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715783    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715785    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715788    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:54.721077 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715790    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715792    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715795    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715797    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715800    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715802    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715805    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715808    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715810    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715813    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715815    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715818    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715820    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715824    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715827    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715831    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715834    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715840    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715842    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715845    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:54.721709 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715848    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715850    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715853    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715856    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715858    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715861    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715863    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715866    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715868    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715871    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715873    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715876    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715878    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715881    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715883    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715886    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715888    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715890    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715893    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715895    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:54.722178 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715898    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715900    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715903    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715905    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715908    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715911    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715915    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715918    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.715943    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715980    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715984    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715988    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.715991    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716007    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716011    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716014    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:54.722704 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716016    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716019    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716022    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716024    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716027    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716030    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716032    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716035    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716037    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716040    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716042    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716045    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716047    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716050    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716052    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716055    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716057    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716060    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716064    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716067    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:54.723080 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716069    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716072    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716075    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716079    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716081    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716085    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716088    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716091    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716094    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716099    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716102    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716104    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716107    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716109    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716112    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716115    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716117    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716120    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716122    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:54.723581 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716125    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716127    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716130    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716132    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716135    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716137    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716140    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716142    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716145    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716147    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716149    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716153    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716156    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716159    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716162    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716165    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716167    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716170    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716175    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716177    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:54.724044 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716180    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716182    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716185    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716187    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716190    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716193    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716195    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716197    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716200    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716202    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716205    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716207    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716210    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716212    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716215    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716217    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716220    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716222    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716225    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:54.724517 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716227    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716230    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716232    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716236    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716242    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716245    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716247    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716250    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716252    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716255    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716258    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716262    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716265    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716268    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716270    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716273    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716275    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716278    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716280    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:54.725031 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716284    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:54.725474 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:54.716286    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:54.725474 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.716351    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 21:45:54.725474 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.719085    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 29 21:45:54.725474 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.720100    2582 server.go:1014] "Starting client certificate rotation"
Oct 29 21:45:54.725474 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.720203    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 21:45:54.725474 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.720246    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 21:45:54.725474 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.722447    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:45:54.725474 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.724678    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:45:54.737255 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.737229    2582 log.go:25] "Validated CRI v1 runtime API"
Oct 29 21:45:54.741579 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.741539    2582 log.go:25] "Validated CRI v1 image API"
Oct 29 21:45:54.742626 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.742601    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 21:45:54.746383 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.746353    2582 fs.go:135] Filesystem UUIDs: map[29e4a222-747d-4d84-b4fd-49720bd6275f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d0dd7a92-3704-452c-b857-d6829f3b88b5:/dev/nvme0n1p3]
Oct 29 21:45:54.746440 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.746381    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 29 21:45:54.753051 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.752848    2582 manager.go:217] Machine: {Timestamp:2026-10-29 21:45:54.751289957 +0000 UTC m=+0.333044994 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100552 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bbb418d50d8eb07d4891f292b865d SystemUUID:ec2bbb41-8d50-d8eb-07d4-891f292b865d BootID:d601cd98-4adb-425d-bb71-90259e43452f Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d2:42:99:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d2:42:99:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:b4:91:d2:34:50 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 21:45:54.753051 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.753041    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 21:45:54.753211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.753155    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 21:45:54.756289 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.756238    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 21:45:54.756500 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.756291    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-38.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 21:45:54.756611 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.756507    2582 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 21:45:54.756611 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.756521    2582 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 21:45:54.756611 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.756598    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 21:45:54.756721 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.756636    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 21:45:54.757943 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.757924    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 21:45:54.762742 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.762716    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 21:45:54.762923 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.762910    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 21:45:54.763797 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.763782    2582 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 21:45:54.763857 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.763804    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 21:45:54.763857 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.763822    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 21:45:54.763961 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.763876    2582 kubelet.go:405] "Adding apiserver pod source"
Oct 29 21:45:54.763961 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.763890    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 21:45:54.764382 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.764361    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 21:45:54.777082 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.776763    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 21:45:54.784053 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.784026    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 21:45:54.784983 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.784969    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 21:45:54.785021 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.785002    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 21:45:54.785211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.785200    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 21:45:54.785241 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.785218    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 21:45:54.785241 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.785225    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 21:45:54.785241 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.785230    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 21:45:54.785241 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.785236    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 21:45:54.785241 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.785242    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 21:45:54.786022 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.786012    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 21:45:54.786086 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.786022    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 21:45:54.786086 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.786030    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 21:45:54.786086 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.786036    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 21:45:54.786086 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.786046    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 21:45:54.786086 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.786055    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 21:45:54.786848 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.786837    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 21:45:54.786891 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.786849    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 21:45:54.790948 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.790927    2582 server.go:1263] "Started kubelet"
Oct 29 21:45:54.791076 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.791039    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 21:45:54.791111 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.791049    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 21:45:54.791150 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.791137    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 21:45:54.791721 ip-10-0-136-38 systemd[1]: Started Kubernetes Kubelet.
Oct 29 21:45:54.792895 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.792844    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 21:45:54.795077 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.795058    2582 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 21:45:54.797836 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:54.797811    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 29 21:45:54.800327 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.800302    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 21:45:54.800942 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.800925    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 21:45:54.801061 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:54.801035    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:54.801746 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.801723    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 21:45:54.801746 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.801728    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 21:45:54.801893 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.801758    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 21:45:54.801893 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.801868    2582 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 21:45:54.801893 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.801876    2582 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 21:45:54.802021 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.802002    2582 factory.go:55] Registering systemd factory
Oct 29 21:45:54.802021 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.802020    2582 factory.go:223] Registration of the systemd container factory successfully
Oct 29 21:45:54.802447 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.802431    2582 factory.go:153] Registering CRI-O factory
Oct 29 21:45:54.802447 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.802450    2582 factory.go:223] Registration of the crio container factory successfully
Oct 29 21:45:54.802586 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.802510    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 21:45:54.802586 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.802530    2582 factory.go:103] Registering Raw factory
Oct 29 21:45:54.802586 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.802587    2582 manager.go:1199] Started watching for new ooms in manager
Oct 29 21:45:54.803845 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.803797    2582 manager.go:319] Starting recovery of all containers
Oct 29 21:45:54.817730 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.817707    2582 manager.go:324] Recovery completed
Oct 29 21:45:54.822001 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.821982    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:54.824042 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.824019    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:54.824155 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.824060    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:54.824155 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.824074    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:54.824709 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.824694    2582 cpu_manager.go:226] "Starting" policy="none"
Oct 29 21:45:54.824791 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.824710    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 21:45:54.824791 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.824733    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 21:45:54.824791 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.824702    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:54.825527 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.825511    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:54.825631 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.825539    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:54.825631 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.825619    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:54.827170 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.827149    2582 policy_none.go:50] "Start"
Oct 29 21:45:54.827255 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.827178    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 21:45:54.827255 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.827196    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 21:45:54.827759 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:54.827727    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 29 21:45:54.829386 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.829366    2582 policy_none.go:44] "Start"
Oct 29 21:45:54.830139 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.830112    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 21:45:54.838324 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.838296    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-38.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 29 21:45:54.839518 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:54.838274    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-38.ec2.internal.18e31e442d572390  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-38.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-38.ec2.internal,},FirstTimestamp:2026-10-29 21:45:54.79088424 +0000 UTC m=+0.372639277,LastTimestamp:2026-10-29 21:45:54.79088424 +0000 UTC m=+0.372639277,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-38.ec2.internal,}"
Oct 29 21:45:54.847388 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:54.847357    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 29 21:45:54.847388 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:54.847351    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 29 21:45:54.847388 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:54.847362    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 29 21:45:54.848832 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.848596    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qdxlf"
Oct 29 21:45:54.859710 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.859684    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qdxlf"
Oct 29 21:45:54.867412 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.867373    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 21:45:54.872096 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.872078    2582 manager.go:341] "Starting Device Plugin manager"
Oct 29 21:45:54.872220 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:54.872122    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 21:45:54.872220 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.872135    2582 server.go:87] "Starting device plugin registration server"
Oct 29 21:45:54.872456 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.872440    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 21:45:54.872516 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.872456    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 21:45:54.872600 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.872580    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 21:45:54.872684 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.872666    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 21:45:54.872684 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.872679    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 21:45:54.873248 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:54.873229    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 21:45:54.873334 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:54.873285    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:54.874962 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:54.874887    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-38.ec2.internal.18e31e442f51119a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-38.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-38.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-38.ec2.internal,},FirstTimestamp:2026-10-29 21:45:54.824040858 +0000 UTC m=+0.405795928,LastTimestamp:2026-10-29 21:45:54.824040858 +0000 UTC m=+0.405795928,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-38.ec2.internal,}"
Oct 29 21:45:54.954032 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.953925    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 21:45:54.956116 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.955467    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 21:45:54.956116 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.955492    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 21:45:54.956116 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.955519    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 21:45:54.956116 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:54.955577    2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 29 21:45:54.969127 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.969099    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:54.973175 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.973148    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:54.974251 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.974225    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:54.974368 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.974264    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:54.974368 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.974277    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:54.974368 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:54.974301    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.014538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.014506    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.014538 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:55.014540    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-38.ec2.internal\": node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:55.056278 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.056238    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"]
Oct 29 21:45:55.056413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.056337    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:55.057369 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.057353    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:55.057439 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.057385    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:55.057439 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.057401    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:55.059671 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.059655    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:55.059831 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.059801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.059939 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.059847    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:55.060758 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.060739    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:55.060851 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.060771    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:55.060851 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.060781    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:55.060851 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.060740    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:55.060851 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.060830    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:55.060851 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.060841    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:55.063078 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.063057    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.063196 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.063095    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:55.063346 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:55.063331    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:55.064595 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.064578    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:55.064713 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.064611    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:55.064713 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.064623    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:55.092598 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:55.092571    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-38.ec2.internal\" not found" node="ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.095303 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:55.095277    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-38.ec2.internal\" not found" node="ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.102684 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.102649    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39a608f7d1c53c952b07789e6ea659e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"39a608f7d1c53c952b07789e6ea659e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.102826 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.102698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39a608f7d1c53c952b07789e6ea659e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"39a608f7d1c53c952b07789e6ea659e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.102826 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.102718    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39cf47f8a2a071911c6009d122ac3420-config\") pod \"kube-apiserver-proxy-ip-10-0-136-38.ec2.internal\" (UID: \"39cf47f8a2a071911c6009d122ac3420\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.164428 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:55.164384    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:55.203796 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.203753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39cf47f8a2a071911c6009d122ac3420-config\") pod \"kube-apiserver-proxy-ip-10-0-136-38.ec2.internal\" (UID: \"39cf47f8a2a071911c6009d122ac3420\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.203796 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.203793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39a608f7d1c53c952b07789e6ea659e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"39a608f7d1c53c952b07789e6ea659e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.204016 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.203813    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39a608f7d1c53c952b07789e6ea659e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"39a608f7d1c53c952b07789e6ea659e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.204016 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.203874    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39a608f7d1c53c952b07789e6ea659e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"39a608f7d1c53c952b07789e6ea659e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.204016 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.203879    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39cf47f8a2a071911c6009d122ac3420-config\") pod \"kube-apiserver-proxy-ip-10-0-136-38.ec2.internal\" (UID: \"39cf47f8a2a071911c6009d122ac3420\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.204016 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.203931    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39a608f7d1c53c952b07789e6ea659e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"39a608f7d1c53c952b07789e6ea659e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.264585 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:55.264482    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:55.365642 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:55.365597    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:55.395830 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.395805    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.397906 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.397882    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:55.466386 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:55.466345    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:55.567080 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:55.566999    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:55.667571 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:55.667513    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:55.720515 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.720477    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 29 21:45:55.720995 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.720694    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 21:45:55.768482 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:55.768442    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:55.795003 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.794969    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:55.865854 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.865767    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 21:40:56 +0000 UTC" deadline="2028-05-24 14:27:15.138903193 +0000 UTC"
Oct 29 21:45:55.865854 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:55.865808    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13744h41m19.273099703s"
Oct 29 21:45:55.869128 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:55.869107    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:55.969880 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:55.969846    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:56.002042 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:56.002005    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:56.003336 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:56.003315    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:56.003480 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:56.003348    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:56.003480 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:56.003357    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:56.039029 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:56.038991    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a608f7d1c53c952b07789e6ea659e9.slice/crio-64227e95b780817a73780d1bd50fe0918a17f696017280aff7ff4d040aed8d6c WatchSource:0}: Error finding container 64227e95b780817a73780d1bd50fe0918a17f696017280aff7ff4d040aed8d6c: Status 404 returned error can't find the container with id 64227e95b780817a73780d1bd50fe0918a17f696017280aff7ff4d040aed8d6c
Oct 29 21:45:56.039466 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:56.039444    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cf47f8a2a071911c6009d122ac3420.slice/crio-95e96df2fcf5249c4d02b0c62687b1782bb6e330bf154c7116adfc74f9765753 WatchSource:0}: Error finding container 95e96df2fcf5249c4d02b0c62687b1782bb6e330bf154c7116adfc74f9765753: Status 404 returned error can't find the container with id 95e96df2fcf5249c4d02b0c62687b1782bb6e330bf154c7116adfc74f9765753
Oct 29 21:45:56.044896 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:56.044875    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 21:45:56.069992 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:56.069950    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:57.443010 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:57.442969    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:57.543686 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:57.543644    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:57.644381 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:57.644334    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:57.730055 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:57.729948    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:57.745078 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:57.745042    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:57.845754 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:57.845692    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:57.946476 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:57.946440    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:45:57.951511 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:57.951480    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:58.008742 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.008659    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:58.048007 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.047977    2582 apiserver.go:52] "Watching apiserver"
Oct 29 21:45:58.073195 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.073161    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 29 21:45:58.073716 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.073693    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:58.077194 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.077171    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 21:45:58.077616 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.077592    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-dmgg8","openshift-ovn-kubernetes/ovnkube-node-pzj9l","kube-system/global-pull-secret-syncer-vth7d","openshift-multus/multus-n4nsq","openshift-network-diagnostics/network-check-target-n8m99","openshift-network-operator/iptables-alerter-l7z5g","kube-system/konnectivity-agent-vwk66","openshift-cluster-node-tuning-operator/tuned-frxl2","openshift-multus/multus-additional-cni-plugins-gd2nd","openshift-multus/network-metrics-daemon-4zxrn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g","openshift-dns/node-resolver-8jkmb"]
Oct 29 21:45:58.080090 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.080043    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:45:58.082910 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.082882    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.085326 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.085305    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:45:58.085430 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.085404    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:45:58.087728 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.087703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.090159 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.090135    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:45:58.090269 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.090211    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:45:58.092485 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.092447    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:45:58.094979 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.094956    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:45:58.097872 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.097849    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:45:58.098003 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.097950    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Oct 29 21:45:58.098382 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.098363    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ckcdt\""
Oct 29 21:45:58.100319 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.098925    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 21:45:58.100319 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.099024    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 21:45:58.100319 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.099054    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 21:45:58.100319 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.099111    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 21:45:58.100319 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.099349    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 21:45:58.100319 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.099379    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 21:45:58.100319 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.099813    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.100319 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.099858    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.101426 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.101406    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 21:45:58.101594 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.101579    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 21:45:58.103938 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.103913    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:45:58.104060 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.104011    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:45:58.104205 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.104186    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6jh7q\""
Oct 29 21:45:58.104509 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.104480    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 21:45:58.105437 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.105409    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 21:45:58.107067 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.107049    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.109515 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.109491    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"]
Oct 29 21:45:58.109659 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.109644    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:45:58.110735 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.110717    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6chnm\""
Oct 29 21:45:58.117867 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.117842    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"]
Oct 29 21:45:58.118145 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.118074    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:45:58.145109 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.144684    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 21:45:58.145109 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.144765    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4t5l4"
Oct 29 21:45:58.175602 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.175569    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 21:45:58.185693 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-cni-binary-copy\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.185862 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09b130d-4249-4539-add0-e0aba69b752a-tmp-dir\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:45:58.185862 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185724    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-env-overrides\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.185862 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-daemon-config\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.185862 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3017c536-859c-4b7b-b59e-f1d5ee77e117-iptables-alerter-script\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:45:58.185862 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-systemd\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.185862 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysctl-conf\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.185862 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-sys\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.185862 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-registration-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.185862 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185853    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-cnibin\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-cni-multus\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185883    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f41e102-fab5-4206-9502-fbf09515069a-serviceca\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185897    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysconfig\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185910    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-system-cni-dir\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-cnibin\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae94d17c-b881-46e1-8063-e88c9dc53369-kubelet-config\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.185994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vrkn\" (UniqueName: \"kubernetes.io/projected/1f41e102-fab5-4206-9502-fbf09515069a-kube-api-access-8vrkn\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-cni-bin\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186040    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-ovnkube-config\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186055    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-k8s-cni-cncf-io\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186077    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vshbg\" (UniqueName: \"kubernetes.io/projected/c2e1c568-867f-472f-9c39-47588d5f3fca-kube-api-access-vshbg\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbtl8\" (UniqueName: \"kubernetes.io/projected/ef2a700a-5061-402a-ab9c-29a44204054c-kube-api-access-tbtl8\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-sys-fs\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-kubelet\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-slash\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.186211 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-kubernetes\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbe53da9-f387-477c-8591-f5c37f42eaf5-tmp\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186213    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae94d17c-b881-46e1-8063-e88c9dc53369-dbus\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186284    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kck4b\" (UniqueName: \"kubernetes.io/projected/f683dbce-ebac-4d2c-ba04-aba3ac20792b-kube-api-access-kck4b\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-multus-certs\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rfd8\" (UniqueName: \"kubernetes.io/projected/801f804c-187a-4232-af07-412a115a2fef-kube-api-access-7rfd8\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-system-cni-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186566    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-hostroot\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186579    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-etc-kubernetes\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/324d72a4-a641-4b09-99d6-d682b5e2c958-konnectivity-ca\") pod \"konnectivity-agent-vwk66\" (UID: \"324d72a4-a641-4b09-99d6-d682b5e2c958\") " pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186608    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-etc-selinux\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.186868 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgf24\" (UniqueName: \"kubernetes.io/projected/a09b130d-4249-4539-add0-e0aba69b752a-kube-api-access-rgf24\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-systemd-units\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-socket-dir-parent\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186712    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-modprobe-d\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-systemd\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186757    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-host\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186782    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-tuned\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6m9n\" (UniqueName: \"kubernetes.io/projected/bbe53da9-f387-477c-8591-f5c37f42eaf5-kube-api-access-v6m9n\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-device-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a09b130d-4249-4539-add0-e0aba69b752a-hosts-file\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-run-netns\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2e1c568-867f-472f-9c39-47588d5f3fca-cni-binary-copy\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.186995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-kubelet\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187019    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-var-lib-kubelet\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht42j\" (UniqueName: \"kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j\") pod \"network-check-target-n8m99\" (UID: \"0fadb982-a597-4489-ae16-3dc4fb5a7810\") " pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:45:58.187470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-run-ovn-kubernetes\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-cni-netd\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-cni-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187142    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3017c536-859c-4b7b-b59e-f1d5ee77e117-host-slash\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187196    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-run\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpbcb\" (UniqueName: \"kubernetes.io/projected/5183b0be-c47a-4e35-a8cb-77991e6f7208-kube-api-access-zpbcb\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187284    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/324d72a4-a641-4b09-99d6-d682b5e2c958-agent-certs\") pod \"konnectivity-agent-vwk66\" (UID: \"324d72a4-a641-4b09-99d6-d682b5e2c958\") " pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187310    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-socket-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-ovn\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187391    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/801f804c-187a-4232-af07-412a115a2fef-ovn-node-metrics-cert\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-node-log\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtbk\" (UniqueName: \"kubernetes.io/projected/3017c536-859c-4b7b-b59e-f1d5ee77e117-kube-api-access-zvtbk\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysctl-d\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-var-lib-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.188125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187555    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-etc-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.188704 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-ovnkube-script-lib\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.188704 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187608    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-cni-bin\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.188704 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-conf-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.188704 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-lib-modules\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.188704 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f41e102-fab5-4206-9502-fbf09515069a-host\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:45:58.188704 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187784    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-log-socket\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.188704 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.187818    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-netns\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.198770 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.198738    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 21:45:58.198938 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.198838    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 21:45:58.198938 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.198897    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 21:45:58.198938 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.198925    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:45:58.199083 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.198974    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 21:45:58.199083 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.198990    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 21:45:58.199189 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199153    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 21:45:58.199245 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199230    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 21:45:58.199296 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199250    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199737    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199743    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mlr8g\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199770    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199780    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199790    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l6fj9\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199824    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199842    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199824    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7dqcg\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199872    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8ld6j\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199899    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199920    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5fns5\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199952    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.199976    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 21:45:58.200102 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.200066    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-znmmx\""
Oct 29 21:45:58.232426 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.232359    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal" event={"ID":"39a608f7d1c53c952b07789e6ea659e9","Type":"ContainerStarted","Data":"64227e95b780817a73780d1bd50fe0918a17f696017280aff7ff4d040aed8d6c"}
Oct 29 21:45:58.234113 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.234072    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal" event={"ID":"39cf47f8a2a071911c6009d122ac3420","Type":"ContainerStarted","Data":"95e96df2fcf5249c4d02b0c62687b1782bb6e330bf154c7116adfc74f9765753"}
Oct 29 21:45:58.288191 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288095    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-hostroot\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.288191 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-etc-kubernetes\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.288191 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288169    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/324d72a4-a641-4b09-99d6-d682b5e2c958-konnectivity-ca\") pod \"konnectivity-agent-vwk66\" (UID: \"324d72a4-a641-4b09-99d6-d682b5e2c958\") " pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:45:58.288191 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-etc-selinux\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgf24\" (UniqueName: \"kubernetes.io/projected/a09b130d-4249-4539-add0-e0aba69b752a-kube-api-access-rgf24\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-systemd-units\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288241    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-socket-dir-parent\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-modprobe-d\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-systemd\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288293    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-host\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-tuned\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6m9n\" (UniqueName: \"kubernetes.io/projected/bbe53da9-f387-477c-8591-f5c37f42eaf5-kube-api-access-v6m9n\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288379    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-device-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a09b130d-4249-4539-add0-e0aba69b752a-hosts-file\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288440    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-run-netns\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2e1c568-867f-472f-9c39-47588d5f3fca-cni-binary-copy\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-kubelet\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.288506 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-var-lib-kubelet\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht42j\" (UniqueName: \"kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j\") pod \"network-check-target-n8m99\" (UID: \"0fadb982-a597-4489-ae16-3dc4fb5a7810\") " pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-run-ovn-kubernetes\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288567    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-cni-netd\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288587    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-cni-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288591    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-systemd\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288608    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3017c536-859c-4b7b-b59e-f1d5ee77e117-host-slash\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288659    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3017c536-859c-4b7b-b59e-f1d5ee77e117-host-slash\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-run\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbcb\" (UniqueName: \"kubernetes.io/projected/5183b0be-c47a-4e35-a8cb-77991e6f7208-kube-api-access-zpbcb\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288724    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-run-netns\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/324d72a4-a641-4b09-99d6-d682b5e2c958-agent-certs\") pod \"konnectivity-agent-vwk66\" (UID: \"324d72a4-a641-4b09-99d6-d682b5e2c958\") " pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288798    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-socket-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288805    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-host\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288821    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-ovn\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288824    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-etc-selinux\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.289178 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/801f804c-187a-4232-af07-412a115a2fef-ovn-node-metrics-cert\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288854    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-cni-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-node-log\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288892    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-device-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288922    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-node-log\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288946    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a09b130d-4249-4539-add0-e0aba69b752a-hosts-file\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288956    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-modprobe-d\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288982    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtbk\" (UniqueName: \"kubernetes.io/projected/3017c536-859c-4b7b-b59e-f1d5ee77e117-kube-api-access-zvtbk\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288994    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288996    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-systemd-units\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289006    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-kubelet\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288718    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-cni-netd\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysctl-d\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289056    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-run\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289102    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-var-lib-kubelet\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289110    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-var-lib-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-etc-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.289987 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289153    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-socket-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-ovnkube-script-lib\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289181    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-ovn\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-cni-bin\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-conf-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289203    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-lib-modules\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f41e102-fab5-4206-9502-fbf09515069a-host\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289275    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289307    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-log-socket\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-netns\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289337    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-run-ovn-kubernetes\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-cni-binary-copy\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289372    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-etc-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09b130d-4249-4539-add0-e0aba69b752a-tmp-dir\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-env-overrides\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289422    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysctl-d\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289432    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-daemon-config\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.290538 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3017c536-859c-4b7b-b59e-f1d5ee77e117-iptables-alerter-script\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-systemd\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysctl-conf\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289548    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-sys\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289571    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-registration-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-cnibin\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289622    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-cni-multus\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289670    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f41e102-fab5-4206-9502-fbf09515069a-serviceca\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysconfig\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289715    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289791    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2e1c568-867f-472f-9c39-47588d5f3fca-cni-binary-copy\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289804    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-system-cni-dir\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-system-cni-dir\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-cnibin\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289853    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-conf-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289149    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-var-lib-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289310    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f41e102-fab5-4206-9502-fbf09515069a-host\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:45:58.291238 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289889    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-cnibin\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289906    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-cni-bin\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289950    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-netns\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289954    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-sys\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289956    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-hostroot\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.289980    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-etc-kubernetes\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae94d17c-b881-46e1-8063-e88c9dc53369-kubelet-config\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrkn\" (UniqueName: \"kubernetes.io/projected/1f41e102-fab5-4206-9502-fbf09515069a-kube-api-access-8vrkn\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-cni-bin\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-ovnkube-config\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290103    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-log-socket\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-k8s-cni-cncf-io\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vshbg\" (UniqueName: \"kubernetes.io/projected/c2e1c568-867f-472f-9c39-47588d5f3fca-kube-api-access-vshbg\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtl8\" (UniqueName: \"kubernetes.io/projected/ef2a700a-5061-402a-ab9c-29a44204054c-kube-api-access-tbtl8\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-sys-fs\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290204    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-kubelet\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-slash\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-kubernetes\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.291976 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbe53da9-f387-477c-8591-f5c37f42eaf5-tmp\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae94d17c-b881-46e1-8063-e88c9dc53369-dbus\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kck4b\" (UniqueName: \"kubernetes.io/projected/f683dbce-ebac-4d2c-ba04-aba3ac20792b-kube-api-access-kck4b\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-multus-certs\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290418    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-daemon-config\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290439    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rfd8\" (UniqueName: \"kubernetes.io/projected/801f804c-187a-4232-af07-412a115a2fef-kube-api-access-7rfd8\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290490    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-registration-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-system-cni-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.290646    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-system-cni-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.291317    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09b130d-4249-4539-add0-e0aba69b752a-tmp-dir\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.291420    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-cni-binary-copy\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.291989    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-ovnkube-script-lib\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.292801 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.292111    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-lib-modules\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.288831    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-socket-dir-parent\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.292384    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-systemd\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.292436    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-slash\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.292448    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-env-overrides\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.292479    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3017c536-859c-4b7b-b59e-f1d5ee77e117-iptables-alerter-script\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.292530    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-kubernetes\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.292709    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-cnibin\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.292739    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-cni-multus\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.292949    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-ovnkube-config\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.293007    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae94d17c-b881-46e1-8063-e88c9dc53369-kubelet-config\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.293010    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.293051    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f41e102-fab5-4206-9502-fbf09515069a-serviceca\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.293082    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae94d17c-b881-46e1-8063-e88c9dc53369-dbus\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.293144    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysconfig\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.293189    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-sys-fs\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.293255    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-kubelet\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.293296    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-k8s-cni-cncf-io\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.293547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.293306    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-multus-certs\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.294425 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.293147    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:58.294425 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.293260    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:58.294425 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.293358    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.294425 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.293373    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysctl-conf\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.294425 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.293403    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-cni-bin\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.294425 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.293443    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret podName:ae94d17c-b881-46e1-8063-e88c9dc53369 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:58.793421524 +0000 UTC m=+3.103095835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret") pod "global-pull-secret-syncer-vth7d" (UID: "ae94d17c-b881-46e1-8063-e88c9dc53369") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:58.294425 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.293315    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.294425 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.293555    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs podName:ef2a700a-5061-402a-ab9c-29a44204054c nodeName:}" failed. No retries permitted until 2026-10-29 21:45:58.793541518 +0000 UTC m=+3.103215840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs") pod "network-metrics-daemon-4zxrn" (UID: "ef2a700a-5061-402a-ab9c-29a44204054c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:58.294788 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.294426    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/324d72a4-a641-4b09-99d6-d682b5e2c958-konnectivity-ca\") pod \"konnectivity-agent-vwk66\" (UID: \"324d72a4-a641-4b09-99d6-d682b5e2c958\") " pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:45:58.294788 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.294739    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/801f804c-187a-4232-af07-412a115a2fef-ovn-node-metrics-cert\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.294956 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.294910    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbe53da9-f387-477c-8591-f5c37f42eaf5-tmp\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.297708 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.297683    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/324d72a4-a641-4b09-99d6-d682b5e2c958-agent-certs\") pod \"konnectivity-agent-vwk66\" (UID: \"324d72a4-a641-4b09-99d6-d682b5e2c958\") " pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:45:58.302241 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.302214    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-tuned\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.310994 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.310959    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:58.310994 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.310999    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:58.310994 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.311014    2582 projected.go:196] Error preparing data for projected volume kube-api-access-ht42j for pod openshift-network-diagnostics/network-check-target-n8m99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:58.311273 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.311098    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j podName:0fadb982-a597-4489-ae16-3dc4fb5a7810 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:58.811076997 +0000 UTC m=+3.120751317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ht42j" (UniqueName: "kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j") pod "network-check-target-n8m99" (UID: "0fadb982-a597-4489-ae16-3dc4fb5a7810") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:58.314343 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.314258    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6m9n\" (UniqueName: \"kubernetes.io/projected/bbe53da9-f387-477c-8591-f5c37f42eaf5-kube-api-access-v6m9n\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.314594 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.314571    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rfd8\" (UniqueName: \"kubernetes.io/projected/801f804c-187a-4232-af07-412a115a2fef-kube-api-access-7rfd8\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.314746 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.314724    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgf24\" (UniqueName: \"kubernetes.io/projected/a09b130d-4249-4539-add0-e0aba69b752a-kube-api-access-rgf24\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:45:58.315610 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.315570    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtbk\" (UniqueName: \"kubernetes.io/projected/3017c536-859c-4b7b-b59e-f1d5ee77e117-kube-api-access-zvtbk\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:45:58.316316 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.316290    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbcb\" (UniqueName: \"kubernetes.io/projected/5183b0be-c47a-4e35-a8cb-77991e6f7208-kube-api-access-zpbcb\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.318570 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.318544    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrkn\" (UniqueName: \"kubernetes.io/projected/1f41e102-fab5-4206-9502-fbf09515069a-kube-api-access-8vrkn\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:45:58.318703 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.318552    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kck4b\" (UniqueName: \"kubernetes.io/projected/f683dbce-ebac-4d2c-ba04-aba3ac20792b-kube-api-access-kck4b\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.318993 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.318971    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbtl8\" (UniqueName: \"kubernetes.io/projected/ef2a700a-5061-402a-ab9c-29a44204054c-kube-api-access-tbtl8\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:45:58.320193 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.320168    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vshbg\" (UniqueName: \"kubernetes.io/projected/c2e1c568-867f-472f-9c39-47588d5f3fca-kube-api-access-vshbg\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.393639 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.393583    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:45:58.405988 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:58.405950    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f41e102_fab5_4206_9502_fbf09515069a.slice/crio-1dd1bcbd88133555363bf351bf334c8198266dea4c32c7054d43127336b3f5cc WatchSource:0}: Error finding container 1dd1bcbd88133555363bf351bf334c8198266dea4c32c7054d43127336b3f5cc: Status 404 returned error can't find the container with id 1dd1bcbd88133555363bf351bf334c8198266dea4c32c7054d43127336b3f5cc
Oct 29 21:45:58.419151 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.419111    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n4nsq"
Oct 29 21:45:58.426487 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.426451    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:45:58.426880 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.426854    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:45:58.429975 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:58.429925    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e1c568_867f_472f_9c39_47588d5f3fca.slice/crio-e01d005eae6c14c03a9634425443b9564b8d5a3e28ded5ea1fc6347a08fec8ba WatchSource:0}: Error finding container e01d005eae6c14c03a9634425443b9564b8d5a3e28ded5ea1fc6347a08fec8ba: Status 404 returned error can't find the container with id e01d005eae6c14c03a9634425443b9564b8d5a3e28ded5ea1fc6347a08fec8ba
Oct 29 21:45:58.431093 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.431066    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:45:58.440143 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.439923    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:45:58.440514 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:58.440477    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3017c536_859c_4b7b_b59e_f1d5ee77e117.slice/crio-d78fcfe0ae169d0d24484aec02afab6b7850f1079511868ac83f9d06b3b590b8 WatchSource:0}: Error finding container d78fcfe0ae169d0d24484aec02afab6b7850f1079511868ac83f9d06b3b590b8: Status 404 returned error can't find the container with id d78fcfe0ae169d0d24484aec02afab6b7850f1079511868ac83f9d06b3b590b8
Oct 29 21:45:58.443774 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:58.442527    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801f804c_187a_4232_af07_412a115a2fef.slice/crio-70b655ab16946107983c75a25d1a564c589aa3d4fc7751290f65f274ca5a4122 WatchSource:0}: Error finding container 70b655ab16946107983c75a25d1a564c589aa3d4fc7751290f65f274ca5a4122: Status 404 returned error can't find the container with id 70b655ab16946107983c75a25d1a564c589aa3d4fc7751290f65f274ca5a4122
Oct 29 21:45:58.445820 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:58.445778    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324d72a4_a641_4b09_99d6_d682b5e2c958.slice/crio-41b90bddf204131ae88a459287592030baa4a2335b039cc08b45c4d34dc96c8f WatchSource:0}: Error finding container 41b90bddf204131ae88a459287592030baa4a2335b039cc08b45c4d34dc96c8f: Status 404 returned error can't find the container with id 41b90bddf204131ae88a459287592030baa4a2335b039cc08b45c4d34dc96c8f
Oct 29 21:45:58.451551 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:58.451518    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe53da9_f387_477c_8591_f5c37f42eaf5.slice/crio-41cb19265accc2269c1c8eb1762f5e3833f47af65ddd5460e15e4f79219c13b4 WatchSource:0}: Error finding container 41cb19265accc2269c1c8eb1762f5e3833f47af65ddd5460e15e4f79219c13b4: Status 404 returned error can't find the container with id 41cb19265accc2269c1c8eb1762f5e3833f47af65ddd5460e15e4f79219c13b4
Oct 29 21:45:58.460790 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.460760    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:45:58.467489 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.467290    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:45:58.470011 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.469985    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:45:58.472508 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:58.472476    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5183b0be_c47a_4e35_a8cb_77991e6f7208.slice/crio-a727e03aa88193020fda425d0065af63e966b5a3dfc59c8264358d9d7dc7c986 WatchSource:0}: Error finding container a727e03aa88193020fda425d0065af63e966b5a3dfc59c8264358d9d7dc7c986: Status 404 returned error can't find the container with id a727e03aa88193020fda425d0065af63e966b5a3dfc59c8264358d9d7dc7c986
Oct 29 21:45:58.479280 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:45:58.479243    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf683dbce_ebac_4d2c_ba04_aba3ac20792b.slice/crio-930a013f9bff036cccddc08345d999cc49d1c9bde3b77806a37c98ec3fc82c98 WatchSource:0}: Error finding container 930a013f9bff036cccddc08345d999cc49d1c9bde3b77806a37c98ec3fc82c98: Status 404 returned error can't find the container with id 930a013f9bff036cccddc08345d999cc49d1c9bde3b77806a37c98ec3fc82c98
Oct 29 21:45:58.795088 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.794954    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:45:58.795088 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.795008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:45:58.795281 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.795127    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:58.795281 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.795200    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret podName:ae94d17c-b881-46e1-8063-e88c9dc53369 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:59.795179449 +0000 UTC m=+4.104853755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret") pod "global-pull-secret-syncer-vth7d" (UID: "ae94d17c-b881-46e1-8063-e88c9dc53369") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:58.795548 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.795131    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:58.795751 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.795592    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs podName:ef2a700a-5061-402a-ab9c-29a44204054c nodeName:}" failed. No retries permitted until 2026-10-29 21:45:59.795574241 +0000 UTC m=+4.105248559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs") pod "network-metrics-daemon-4zxrn" (UID: "ef2a700a-5061-402a-ab9c-29a44204054c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:58.896564 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:58.896254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht42j\" (UniqueName: \"kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j\") pod \"network-check-target-n8m99\" (UID: \"0fadb982-a597-4489-ae16-3dc4fb5a7810\") " pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:45:58.896826 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.896793    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:58.896826 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.896820    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:58.896948 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.896833    2582 projected.go:196] Error preparing data for projected volume kube-api-access-ht42j for pod openshift-network-diagnostics/network-check-target-n8m99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:58.896948 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:58.896901    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j podName:0fadb982-a597-4489-ae16-3dc4fb5a7810 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:59.896883537 +0000 UTC m=+4.206557847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ht42j" (UniqueName: "kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j") pod "network-check-target-n8m99" (UID: "0fadb982-a597-4489-ae16-3dc4fb5a7810") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:59.148284 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.148212    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:40:58 +0000 UTC" deadline="2028-06-13 10:12:10.459652814 +0000 UTC"
Oct 29 21:45:59.148284 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.148278    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14220h26m11.311378751s"
Oct 29 21:45:59.229151 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.228621    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:45:59.229151 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:59.228777    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:45:59.244163 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.244035    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2nd" event={"ID":"5183b0be-c47a-4e35-a8cb-77991e6f7208","Type":"ContainerStarted","Data":"a727e03aa88193020fda425d0065af63e966b5a3dfc59c8264358d9d7dc7c986"}
Oct 29 21:45:59.246166 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.246086    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-frxl2" event={"ID":"bbe53da9-f387-477c-8591-f5c37f42eaf5","Type":"ContainerStarted","Data":"41cb19265accc2269c1c8eb1762f5e3833f47af65ddd5460e15e4f79219c13b4"}
Oct 29 21:45:59.249793 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.249696    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vwk66" event={"ID":"324d72a4-a641-4b09-99d6-d682b5e2c958","Type":"ContainerStarted","Data":"41b90bddf204131ae88a459287592030baa4a2335b039cc08b45c4d34dc96c8f"}
Oct 29 21:45:59.257250 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.257095    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l" event={"ID":"801f804c-187a-4232-af07-412a115a2fef","Type":"ContainerStarted","Data":"70b655ab16946107983c75a25d1a564c589aa3d4fc7751290f65f274ca5a4122"}
Oct 29 21:45:59.260290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.260192    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l7z5g" event={"ID":"3017c536-859c-4b7b-b59e-f1d5ee77e117","Type":"ContainerStarted","Data":"d78fcfe0ae169d0d24484aec02afab6b7850f1079511868ac83f9d06b3b590b8"}
Oct 29 21:45:59.264679 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.264557    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g" event={"ID":"f683dbce-ebac-4d2c-ba04-aba3ac20792b","Type":"ContainerStarted","Data":"930a013f9bff036cccddc08345d999cc49d1c9bde3b77806a37c98ec3fc82c98"}
Oct 29 21:45:59.266986 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.266910    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n4nsq" event={"ID":"c2e1c568-867f-472f-9c39-47588d5f3fca","Type":"ContainerStarted","Data":"e01d005eae6c14c03a9634425443b9564b8d5a3e28ded5ea1fc6347a08fec8ba"}
Oct 29 21:45:59.269353 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.269307    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dmgg8" event={"ID":"1f41e102-fab5-4206-9502-fbf09515069a","Type":"ContainerStarted","Data":"1dd1bcbd88133555363bf351bf334c8198266dea4c32c7054d43127336b3f5cc"}
Oct 29 21:45:59.272474 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.272442    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal" event={"ID":"39cf47f8a2a071911c6009d122ac3420","Type":"ContainerStarted","Data":"710ddbcfc63f91fae46a6a3e95cb1b47fbda598be2d45ab724c451183516fd9a"}
Oct 29 21:45:59.278285 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.278253    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jkmb" event={"ID":"a09b130d-4249-4539-add0-e0aba69b752a","Type":"ContainerStarted","Data":"2deaa56200d86b0a245d5d869577bf52e61f8ae0b923ff824368b1099a340f41"}
Oct 29 21:45:59.809316 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.809262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:45:59.809316 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.809312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:45:59.809547 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:59.809427    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:59.809547 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:59.809474    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs podName:ef2a700a-5061-402a-ab9c-29a44204054c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:01.809461217 +0000 UTC m=+6.119135521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs") pod "network-metrics-daemon-4zxrn" (UID: "ef2a700a-5061-402a-ab9c-29a44204054c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:59.809806 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:59.809790    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:59.809872 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:59.809830    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret podName:ae94d17c-b881-46e1-8063-e88c9dc53369 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:01.809819899 +0000 UTC m=+6.119494203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret") pod "global-pull-secret-syncer-vth7d" (UID: "ae94d17c-b881-46e1-8063-e88c9dc53369") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:59.909945 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:45:59.909854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht42j\" (UniqueName: \"kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j\") pod \"network-check-target-n8m99\" (UID: \"0fadb982-a597-4489-ae16-3dc4fb5a7810\") " pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:45:59.910111 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:59.910046    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:59.910111 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:59.910069    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:59.910111 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:59.910083    2582 projected.go:196] Error preparing data for projected volume kube-api-access-ht42j for pod openshift-network-diagnostics/network-check-target-n8m99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:59.910233 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:45:59.910141    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j podName:0fadb982-a597-4489-ae16-3dc4fb5a7810 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:01.910123551 +0000 UTC m=+6.219797870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ht42j" (UniqueName: "kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j") pod "network-check-target-n8m99" (UID: "0fadb982-a597-4489-ae16-3dc4fb5a7810") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:46:00.149250 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:00.149178    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:40:58 +0000 UTC" deadline="2028-04-13 09:02:02.161876698 +0000 UTC"
Oct 29 21:46:00.149250 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:00.149219    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12755h16m2.012662402s"
Oct 29 21:46:00.231470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:00.230647    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:00.231470 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:00.230785    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:00.231470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:00.231208    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:00.231470 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:00.231323    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:00.294069 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:00.293873    2582 generic.go:356] "Generic (PLEG): container finished" podID="39a608f7d1c53c952b07789e6ea659e9" containerID="499d23babcc83b7dcc144bbcc9f1e5a27350c8da01f29a9799c6e606b4d61010" exitCode=0
Oct 29 21:46:00.294069 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:00.293924    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal" event={"ID":"39a608f7d1c53c952b07789e6ea659e9","Type":"ContainerDied","Data":"499d23babcc83b7dcc144bbcc9f1e5a27350c8da01f29a9799c6e606b4d61010"}
Oct 29 21:46:00.337235 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:00.336574    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal" podStartSLOduration=2.336552629 podStartE2EDuration="2.336552629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:59.314247595 +0000 UTC m=+3.623921922" watchObservedRunningTime="2026-10-29 21:46:00.336552629 +0000 UTC m=+4.646227028"
Oct 29 21:46:01.228750 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:01.228714    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:01.229175 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:01.228858    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:01.308968 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:01.308931    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal" event={"ID":"39a608f7d1c53c952b07789e6ea659e9","Type":"ContainerStarted","Data":"1b36b2348f6eed576a2e5a5bb9d950b33de393beb0004c441d829efc84602b86"}
Oct 29 21:46:01.319374 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:01.319331    2582 generic.go:356] "Generic (PLEG): container finished" podID="5183b0be-c47a-4e35-a8cb-77991e6f7208" containerID="e79587e333bee33eec85d67583efa01672a06204efa0c5fee0b048ced5d32a9c" exitCode=0
Oct 29 21:46:01.319546 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:01.319389    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2nd" event={"ID":"5183b0be-c47a-4e35-a8cb-77991e6f7208","Type":"ContainerDied","Data":"e79587e333bee33eec85d67583efa01672a06204efa0c5fee0b048ced5d32a9c"}
Oct 29 21:46:01.354647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:01.354575    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal" podStartSLOduration=3.354543582 podStartE2EDuration="3.354543582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:01.353712527 +0000 UTC m=+5.663386854" watchObservedRunningTime="2026-10-29 21:46:01.354543582 +0000 UTC m=+5.664217910"
Oct 29 21:46:01.829240 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:01.829194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:01.829422 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:01.829248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:01.829422 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:01.829397    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:46:01.829531 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:01.829463    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs podName:ef2a700a-5061-402a-ab9c-29a44204054c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:05.829445623 +0000 UTC m=+10.139119942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs") pod "network-metrics-daemon-4zxrn" (UID: "ef2a700a-5061-402a-ab9c-29a44204054c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:46:01.829888 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:01.829837    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:46:01.830000 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:01.829895    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret podName:ae94d17c-b881-46e1-8063-e88c9dc53369 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:05.829878502 +0000 UTC m=+10.139552814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret") pod "global-pull-secret-syncer-vth7d" (UID: "ae94d17c-b881-46e1-8063-e88c9dc53369") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:46:01.930589 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:01.930546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht42j\" (UniqueName: \"kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j\") pod \"network-check-target-n8m99\" (UID: \"0fadb982-a597-4489-ae16-3dc4fb5a7810\") " pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:01.930868 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:01.930848    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:46:01.930942 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:01.930875    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:46:01.930942 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:01.930924    2582 projected.go:196] Error preparing data for projected volume kube-api-access-ht42j for pod openshift-network-diagnostics/network-check-target-n8m99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:46:01.931065 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:01.930986    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j podName:0fadb982-a597-4489-ae16-3dc4fb5a7810 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:05.930967267 +0000 UTC m=+10.240641577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ht42j" (UniqueName: "kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j") pod "network-check-target-n8m99" (UID: "0fadb982-a597-4489-ae16-3dc4fb5a7810") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:46:02.230443 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:02.230408    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:02.231049 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:02.230566    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:02.231049 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:02.231003    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:02.231178 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:02.231105    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:02.335406 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:02.335305    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vwk66" event={"ID":"324d72a4-a641-4b09-99d6-d682b5e2c958","Type":"ContainerStarted","Data":"6ce8ffe422185b028a8317a43a89a597322ab8a7de0c859d5b9796261fb5688d"}
Oct 29 21:46:02.377385 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:02.377310    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vwk66" podStartSLOduration=2.764422107 podStartE2EDuration="6.377287479s" totalImagesPullingTime="3.612865372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:02.376355566 +0000 UTC m=+6.686029905" watchObservedRunningTime="2026-10-29 21:46:02.377287479 +0000 UTC m=+6.686961806"
Oct 29 21:46:03.227883 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:03.227842    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:03.228107 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:03.228004    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:03.653854 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:03.653811    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:46:03.654430 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:03.654407    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:46:04.229137 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:04.228580    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:04.229137 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:04.228764    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:04.229737 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:04.229562    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:04.229737 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:04.229688    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:04.349893 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:04.349848    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g" event={"ID":"f683dbce-ebac-4d2c-ba04-aba3ac20792b","Type":"ContainerStarted","Data":"182766c25c198c5de5c44d0383c2cac62e63062c13ac62729701fd890ab3d493"}
Oct 29 21:46:04.353322 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:04.353051    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dmgg8" event={"ID":"1f41e102-fab5-4206-9502-fbf09515069a","Type":"ContainerStarted","Data":"a42e03d93cfbd176b0e60d8427852bc0b5f6dfe3bff386451f7f4138833987a2"}
Oct 29 21:46:04.353322 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:04.353113    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:46:04.354041 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:04.353995    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:46:04.395548 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:04.395469    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dmgg8" podStartSLOduration=3.406470022 podStartE2EDuration="8.395447245s" totalImagesPullingTime="4.988977223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:04.394550213 +0000 UTC m=+8.704224541" watchObservedRunningTime="2026-10-29 21:46:04.395447245 +0000 UTC m=+8.705121582"
Oct 29 21:46:05.227862 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:05.227816    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:05.228267 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:05.227977    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:05.868675 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:05.867842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:05.868675 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:05.867970    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:05.868675 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:05.868116    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:46:05.868675 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:05.868182    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret podName:ae94d17c-b881-46e1-8063-e88c9dc53369 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:13.868164245 +0000 UTC m=+18.177838550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret") pod "global-pull-secret-syncer-vth7d" (UID: "ae94d17c-b881-46e1-8063-e88c9dc53369") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:46:05.868675 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:05.868600    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:46:05.868675 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:05.868673    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs podName:ef2a700a-5061-402a-ab9c-29a44204054c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:13.868658121 +0000 UTC m=+18.178332438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs") pod "network-metrics-daemon-4zxrn" (UID: "ef2a700a-5061-402a-ab9c-29a44204054c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:46:05.969199 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:05.969164    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht42j\" (UniqueName: \"kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j\") pod \"network-check-target-n8m99\" (UID: \"0fadb982-a597-4489-ae16-3dc4fb5a7810\") " pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:05.969368 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:05.969326    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:46:05.969368 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:05.969341    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:46:05.969368 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:05.969349    2582 projected.go:196] Error preparing data for projected volume kube-api-access-ht42j for pod openshift-network-diagnostics/network-check-target-n8m99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:46:05.969513 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:05.969407    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j podName:0fadb982-a597-4489-ae16-3dc4fb5a7810 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:13.969389882 +0000 UTC m=+18.279064187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ht42j" (UniqueName: "kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j") pod "network-check-target-n8m99" (UID: "0fadb982-a597-4489-ae16-3dc4fb5a7810") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:46:05.980751 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:05.980670    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 21:46:06.156845 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:06.155996    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T21:46:05.980723982Z","UUID":"4c23e878-c8bd-4fcf-968b-d5fd2328fb8d","Handler":null,"Name":"","Endpoint":""}
Oct 29 21:46:06.158251 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:06.158225    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 21:46:06.158251 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:06.158263    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 21:46:06.229503 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:06.229079    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:06.229503 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:06.229191    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:06.229503 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:06.229499    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:06.230077 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:06.229560    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:06.359710 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:06.359611    2582 generic.go:356] "Generic (PLEG): container finished" podID="5183b0be-c47a-4e35-a8cb-77991e6f7208" containerID="529a352b49a16fc1999791c8190b7cfa9cb16b8e8d33b2da1d1db4965960d4bf" exitCode=0
Oct 29 21:46:06.359880 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:06.359766    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2nd" event={"ID":"5183b0be-c47a-4e35-a8cb-77991e6f7208","Type":"ContainerDied","Data":"529a352b49a16fc1999791c8190b7cfa9cb16b8e8d33b2da1d1db4965960d4bf"}
Oct 29 21:46:06.363993 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:06.363814    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g" event={"ID":"f683dbce-ebac-4d2c-ba04-aba3ac20792b","Type":"ContainerStarted","Data":"67adfb1467f3fb7ff5c13f83854518e954d9caa3c4b7fbff28b372d37a05ce04"}
Oct 29 21:46:06.366971 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:06.366887    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n4nsq" event={"ID":"c2e1c568-867f-472f-9c39-47588d5f3fca","Type":"ContainerStarted","Data":"6d720654d0045d8acaa1469ae6a1c607e1e981dd2dee3ef1c4dc6c2d2dd1237b"}
Oct 29 21:46:06.458289 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:06.458221    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-n4nsq" podStartSLOduration=2.653028716 podStartE2EDuration="10.458198992s" totalImagesPullingTime="7.805170276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:06.457307169 +0000 UTC m=+10.766981498" watchObservedRunningTime="2026-10-29 21:46:06.458198992 +0000 UTC m=+10.767873321"
Oct 29 21:46:07.228938 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:07.228685    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:07.229104 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:07.229046    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:07.372013 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:07.371974    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g" event={"ID":"f683dbce-ebac-4d2c-ba04-aba3ac20792b","Type":"ContainerStarted","Data":"fbd089831314b50c49a7a680988bbcce27ddb0e0e8bb906ac89df1a4a244b695"}
Oct 29 21:46:07.373960 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:07.373926    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8jkmb" event={"ID":"a09b130d-4249-4539-add0-e0aba69b752a","Type":"ContainerStarted","Data":"091eb83b3b40cdf365c90dec20f52c231f8a3b78287bf95f90a16bff7f7c85a5"}
Oct 29 21:46:07.377157 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:07.377122    2582 generic.go:356] "Generic (PLEG): container finished" podID="5183b0be-c47a-4e35-a8cb-77991e6f7208" containerID="a9b10383bc398569e9cbf1f28287ea8c2e1db59f64ec81c568fa1c925cb1c774" exitCode=0
Oct 29 21:46:07.378186 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:07.378158    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2nd" event={"ID":"5183b0be-c47a-4e35-a8cb-77991e6f7208","Type":"ContainerDied","Data":"a9b10383bc398569e9cbf1f28287ea8c2e1db59f64ec81c568fa1c925cb1c774"}
Oct 29 21:46:07.423334 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:07.422005    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g" podStartSLOduration=3.212507772 podStartE2EDuration="11.42198401s" totalImagesPullingTime="8.209476238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:07.420866056 +0000 UTC m=+11.730540387" watchObservedRunningTime="2026-10-29 21:46:07.42198401 +0000 UTC m=+11.731658338"
Oct 29 21:46:08.228374 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:08.228343    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:08.228540 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:08.228505    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:08.229044 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:08.228812    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:08.229044 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:08.228921    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:08.383064 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:08.383012    2582 generic.go:356] "Generic (PLEG): container finished" podID="5183b0be-c47a-4e35-a8cb-77991e6f7208" containerID="02fe1cc540f7281ac0e2aafbbdcea73a4042bf472b07c24ad527061378e60df8" exitCode=0
Oct 29 21:46:08.383541 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:08.383117    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2nd" event={"ID":"5183b0be-c47a-4e35-a8cb-77991e6f7208","Type":"ContainerDied","Data":"02fe1cc540f7281ac0e2aafbbdcea73a4042bf472b07c24ad527061378e60df8"}
Oct 29 21:46:08.385521 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:08.385325    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l7z5g" event={"ID":"3017c536-859c-4b7b-b59e-f1d5ee77e117","Type":"ContainerStarted","Data":"44e28f491d6972bc7df6641881d2bd75870ead37dd34b341856d6cafe425d0ba"}
Oct 29 21:46:08.432483 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:08.432426    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-8jkmb" podStartSLOduration=4.371683515 podStartE2EDuration="12.432410453s" totalImagesPullingTime="8.060726938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:07.519700029 +0000 UTC m=+11.829374355" watchObservedRunningTime="2026-10-29 21:46:08.432410453 +0000 UTC m=+12.742084778"
Oct 29 21:46:08.479071 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:08.478990    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l7z5g" podStartSLOduration=4.536795151 podStartE2EDuration="12.478970825s" totalImagesPullingTime="7.942175674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:08.478769115 +0000 UTC m=+12.788443442" watchObservedRunningTime="2026-10-29 21:46:08.478970825 +0000 UTC m=+12.788645153"
Oct 29 21:46:09.228857 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:09.228819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:09.229040 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:09.228973    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:09.392836 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:09.392784    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-frxl2" event={"ID":"bbe53da9-f387-477c-8591-f5c37f42eaf5","Type":"ContainerStarted","Data":"88af1d41641c0cbce11d15a157ccd5ea7a1ffae2684a538e47635d702b6fe275"}
Oct 29 21:46:09.439391 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:09.438951    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-frxl2" podStartSLOduration=3.099754376 podStartE2EDuration="13.438931449s" totalImagesPullingTime="10.339177073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:09.438337968 +0000 UTC m=+13.748012295" watchObservedRunningTime="2026-10-29 21:46:09.438931449 +0000 UTC m=+13.748605778"
Oct 29 21:46:10.228045 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:10.228002    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:10.228045 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:10.228033    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:10.228296 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:10.228170    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:10.228296 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:10.228279    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:11.228457 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:11.228417    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:11.229171 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:11.228576    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:11.400542 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:11.400499    2582 generic.go:356] "Generic (PLEG): container finished" podID="5183b0be-c47a-4e35-a8cb-77991e6f7208" containerID="308bc48fe717d9352916673b8ab574dbe54e44835836425fecd63d66afdf8fb6" exitCode=0
Oct 29 21:46:11.400735 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:11.400585    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2nd" event={"ID":"5183b0be-c47a-4e35-a8cb-77991e6f7208","Type":"ContainerDied","Data":"308bc48fe717d9352916673b8ab574dbe54e44835836425fecd63d66afdf8fb6"}
Oct 29 21:46:12.228762 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:12.228474    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:12.229242 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:12.228514    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:12.229242 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:12.228899    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:12.229242 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:12.228988    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:12.407114 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:12.407077    2582 generic.go:356] "Generic (PLEG): container finished" podID="5183b0be-c47a-4e35-a8cb-77991e6f7208" containerID="beb1209d08fa4d6dd6029800dc6cd4f31cf27bfbc3216b7a4b4d114794217a7c" exitCode=0
Oct 29 21:46:12.407319 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:12.407150    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2nd" event={"ID":"5183b0be-c47a-4e35-a8cb-77991e6f7208","Type":"ContainerDied","Data":"beb1209d08fa4d6dd6029800dc6cd4f31cf27bfbc3216b7a4b4d114794217a7c"}
Oct 29 21:46:13.228092 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:13.227875    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:13.228092 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:13.228037    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:13.412921 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:13.412889    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2nd" event={"ID":"5183b0be-c47a-4e35-a8cb-77991e6f7208","Type":"ContainerStarted","Data":"3d0460497ad1bccc05b05ed45b6c1586d0fea33dadf4adae54e938f713ad3eb3"}
Oct 29 21:46:13.415236 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:13.415209    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l" event={"ID":"801f804c-187a-4232-af07-412a115a2fef","Type":"ContainerStarted","Data":"708fea0cc2629f1d03ce3d8282370c2e6f1b747c2c3dee1fa6f426c5fff5b6ab"}
Oct 29 21:46:13.415363 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:13.415279    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l" event={"ID":"801f804c-187a-4232-af07-412a115a2fef","Type":"ContainerStarted","Data":"0608894a7cf11690195c402114c999c695899fa25bb7faa31fec2359fa58abb1"}
Oct 29 21:46:13.415363 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:13.415308    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l" event={"ID":"801f804c-187a-4232-af07-412a115a2fef","Type":"ContainerStarted","Data":"5424cb82c78a62c5eae2d3197998806620d698168744a66d57b4b57b5e6d1f1c"}
Oct 29 21:46:13.415363 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:13.415336    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l" event={"ID":"801f804c-187a-4232-af07-412a115a2fef","Type":"ContainerStarted","Data":"1f037e0542432a97a3ca322b1d612258d55ca274dd87216f3f793a30d12a53b4"}
Oct 29 21:46:13.939253 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:13.939214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:13.939253 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:13.939250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:13.939460 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:13.939370    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:46:13.939460 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:13.939375    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:46:13.939460 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:13.939440    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs podName:ef2a700a-5061-402a-ab9c-29a44204054c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:29.939421529 +0000 UTC m=+34.249095853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs") pod "network-metrics-daemon-4zxrn" (UID: "ef2a700a-5061-402a-ab9c-29a44204054c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:46:13.939460 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:13.939458    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret podName:ae94d17c-b881-46e1-8063-e88c9dc53369 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:29.93944913 +0000 UTC m=+34.249123441 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret") pod "global-pull-secret-syncer-vth7d" (UID: "ae94d17c-b881-46e1-8063-e88c9dc53369") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:46:14.040113 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:14.040075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht42j\" (UniqueName: \"kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j\") pod \"network-check-target-n8m99\" (UID: \"0fadb982-a597-4489-ae16-3dc4fb5a7810\") " pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:14.040267 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:14.040248    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:46:14.040321 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:14.040278    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:46:14.040321 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:14.040292    2582 projected.go:196] Error preparing data for projected volume kube-api-access-ht42j for pod openshift-network-diagnostics/network-check-target-n8m99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:46:14.040374 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:14.040354    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j podName:0fadb982-a597-4489-ae16-3dc4fb5a7810 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:30.040337328 +0000 UTC m=+34.350011634 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ht42j" (UniqueName: "kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j") pod "network-check-target-n8m99" (UID: "0fadb982-a597-4489-ae16-3dc4fb5a7810") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:46:14.228413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:14.228326    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:14.228565 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:14.228499    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:14.228565 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:14.228534    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:14.228702 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:14.228615    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:14.420938 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:14.420900    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l" event={"ID":"801f804c-187a-4232-af07-412a115a2fef","Type":"ContainerStarted","Data":"d4a0050632857fe842e9f1d8014f6382412adbf8be90d66201761e7dc927d233"}
Oct 29 21:46:14.421326 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:14.420992    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l" event={"ID":"801f804c-187a-4232-af07-412a115a2fef","Type":"ContainerStarted","Data":"d7b2569401dece262d553b5cafdce8cbc3ea77944f81c8993251bbf4f23ceced"}
Oct 29 21:46:15.228207 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:15.228165    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:15.228369 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:15.228272    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:16.229104 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:16.229062    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:16.229656 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:16.229117    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:16.229656 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:16.229206    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:16.229656 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:16.229306    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:16.428649 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:16.428589    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l" event={"ID":"801f804c-187a-4232-af07-412a115a2fef","Type":"ContainerStarted","Data":"566850c832da163397a5f984fa874d57f5313e5e1499ff6dd591c6f8bc9fcedb"}
Oct 29 21:46:17.228603 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:17.228570    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:17.228788 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:17.228711    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:18.228317 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:18.228279    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:18.229212 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:18.228283    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:18.229212 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:18.228437    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:18.229212 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:18.228549    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:18.437173 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:18.437133    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l" event={"ID":"801f804c-187a-4232-af07-412a115a2fef","Type":"ContainerStarted","Data":"aa1f1f16dbfed412da53e6ed2fccce38d6a6a3e76743e4b8ac9779e77e119612"}
Oct 29 21:46:18.437498 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:18.437480    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:46:18.437559 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:18.437550    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:46:18.454817 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:18.454787    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:46:18.530977 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:18.530877    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l" podStartSLOduration=7.8428356919999995 podStartE2EDuration="22.530845664s" totalImagesPullingTime="14.688009972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:18.518040094 +0000 UTC m=+22.827714420" watchObservedRunningTime="2026-10-29 21:46:18.530845664 +0000 UTC m=+22.840519981"
Oct 29 21:46:18.531334 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:18.531312    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gd2nd" podStartSLOduration=12.162054312 podStartE2EDuration="22.531304565s" totalImagesPullingTime="10.369250253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:13.471393074 +0000 UTC m=+17.781067402" watchObservedRunningTime="2026-10-29 21:46:18.531304565 +0000 UTC m=+22.840978892"
Oct 29 21:46:19.228373 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:19.228338    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:19.228858 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:19.228444    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:19.440277 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:19.440248    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:46:19.456934 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:19.456902    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:46:20.228902 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:20.228862    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:20.229299 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:20.228988    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:20.229299 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:20.229041    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:20.229299 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:20.229132    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:21.227816 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:21.227780    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:21.227971 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:21.227886    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:22.228151 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:22.228117    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:22.228585 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:22.228117    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:22.228585 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:22.228240    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:22.228585 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:22.228292    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:23.228555 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:23.228519    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:23.229064 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:23.228653    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:24.231309 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:24.231282    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:24.231781 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:24.231292    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:24.231781 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:24.231376    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:24.231781 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:24.231449    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:25.227852 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:25.227811    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:25.228034 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:25.227928    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:26.229079 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:26.228866    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:26.229558 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:26.228934    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:26.229558 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:26.229141    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:26.229558 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:26.229243    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:27.228296 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:27.228262    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:27.228477 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:27.228370    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:28.228991 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:28.228948    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:28.229427 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:28.229002    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:28.229427 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:28.229110    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:28.229427 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:28.229208    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:29.228396 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:29.228356    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:29.228574 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:29.228463    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:29.953679 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:29.953617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:29.954075 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:29.953690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:29.954075 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:29.953780    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:46:29.954075 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:29.953843    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret podName:ae94d17c-b881-46e1-8063-e88c9dc53369 nodeName:}" failed. No retries permitted until 2026-10-29 21:47:01.953829081 +0000 UTC m=+66.263503398 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret") pod "global-pull-secret-syncer-vth7d" (UID: "ae94d17c-b881-46e1-8063-e88c9dc53369") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:46:29.954075 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:29.953841    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:46:29.954075 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:29.953901    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs podName:ef2a700a-5061-402a-ab9c-29a44204054c nodeName:}" failed. No retries permitted until 2026-10-29 21:47:01.953881684 +0000 UTC m=+66.263555989 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs") pod "network-metrics-daemon-4zxrn" (UID: "ef2a700a-5061-402a-ab9c-29a44204054c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:46:30.054893 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:30.054854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht42j\" (UniqueName: \"kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j\") pod \"network-check-target-n8m99\" (UID: \"0fadb982-a597-4489-ae16-3dc4fb5a7810\") " pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:30.055066 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:30.055003    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:46:30.055066 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:30.055018    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:46:30.055066 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:30.055031    2582 projected.go:196] Error preparing data for projected volume kube-api-access-ht42j for pod openshift-network-diagnostics/network-check-target-n8m99: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:46:30.055159 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:30.055084    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j podName:0fadb982-a597-4489-ae16-3dc4fb5a7810 nodeName:}" failed. No retries permitted until 2026-10-29 21:47:02.055069064 +0000 UTC m=+66.364743371 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ht42j" (UniqueName: "kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j") pod "network-check-target-n8m99" (UID: "0fadb982-a597-4489-ae16-3dc4fb5a7810") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:46:30.228328 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:30.228211    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:30.228328 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:30.228258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:30.228501 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:30.228349    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:30.228501 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:30.228454    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:31.227990 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:31.227953    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:31.228407 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:31.228078    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:32.228569 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:32.228531    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:32.229077 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:32.228672    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:32.229077 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:32.228702    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:32.229077 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:32.228750    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:33.228154 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:33.228119    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:33.228331 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:33.228235    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:34.228326 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:34.228288    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:34.228767 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:34.228427    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:34.228767 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:34.228475    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:34.228767 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:34.228569    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:35.227994 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:35.227954    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:35.228187 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:35.228077    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:36.230828 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:36.230787    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:36.231551 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:36.231519    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:36.231946 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:36.231926    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:36.232479 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:36.232449    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:37.228848 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:37.228804    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:37.229070 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:37.228940    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:38.228933 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:38.228894    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:38.229350 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:38.228903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:38.229350 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:38.229026    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:38.229350 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:38.229091    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:38.788168 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:38.788081    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n8m99"]
Oct 29 21:46:38.788368 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:38.788218    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:38.788368 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:38.788334    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:38.792000 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:38.791964    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vth7d"]
Oct 29 21:46:38.792327 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:38.792098    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:38.792327 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:38.792212    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:38.792673 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:38.792652    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4zxrn"]
Oct 29 21:46:38.792748 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:38.792740    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:38.792841 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:38.792824    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:39.939886 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:39.939851    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8jkmb_a09b130d-4249-4539-add0-e0aba69b752a/dns-node-resolver/0.log"
Oct 29 21:46:40.228872 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:40.228830    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:40.229051 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:40.228845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:40.229051 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:40.228956    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:40.229051 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:40.229013    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:40.229220 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:40.229111    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:40.229220 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:40.229164    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:40.874696 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:40.874662    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dmgg8_1f41e102-fab5-4206-9502-fbf09515069a/node-ca/0.log"
Oct 29 21:46:42.228580 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:42.228539    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:42.229112 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:42.228589    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:42.229112 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:42.228681    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:42.229112 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:42.228771    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n8m99" podUID="0fadb982-a597-4489-ae16-3dc4fb5a7810"
Oct 29 21:46:42.229112 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:42.228906    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4zxrn" podUID="ef2a700a-5061-402a-ab9c-29a44204054c"
Oct 29 21:46:42.229112 ip-10-0-136-38 kubenswrapper[2582]: E1029 21:46:42.228983    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vth7d" podUID="ae94d17c-b881-46e1-8063-e88c9dc53369"
Oct 29 21:46:42.922253 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:42.922215    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeReady"
Oct 29 21:46:42.922432 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:42.922345    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 29 21:46:42.989945 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:42.989858    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2867l"]
Oct 29 21:46:42.991609 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:42.991581    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:42.993182 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:42.993153    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-245p6"]
Oct 29 21:46:42.994891 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:42.994869    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.007760 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.007734    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w9rdw\""
Oct 29 21:46:43.007760 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.007734    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 21:46:43.007924 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.007761    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 21:46:43.007924 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.007734    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 21:46:43.007924 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.007775    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 21:46:43.010235 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.010215    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 21:46:43.010334 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.010237    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vdqjz\""
Oct 29 21:46:43.021119 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.021052    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2867l"]
Oct 29 21:46:43.021928 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.021900    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-245p6"]
Oct 29 21:46:43.106185 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.106148    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zv5gd"]
Oct 29 21:46:43.107843 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.107827    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zv5gd"
Oct 29 21:46:43.108460 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.108437    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 21:46:43.123948 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.123920    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 21:46:43.124125 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.124041    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 21:46:43.131805 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.131768    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zv5gd"]
Oct 29 21:46:43.148775 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.148730    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79bf4395-e3d6-4071-aedd-548c185fa39d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.148977 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.148792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79bf4395-e3d6-4071-aedd-548c185fa39d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.148977 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.148891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3accc688-8a04-4462-9593-228a18548478-config-volume\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.148977 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.148942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3accc688-8a04-4462-9593-228a18548478-tmp-dir\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.148977 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.148969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnclr\" (UniqueName: \"kubernetes.io/projected/3accc688-8a04-4462-9593-228a18548478-kube-api-access-gnclr\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.149134 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.149025    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmrsr\" (UniqueName: \"kubernetes.io/projected/79bf4395-e3d6-4071-aedd-548c185fa39d-kube-api-access-kmrsr\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.149134 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.149082    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3accc688-8a04-4462-9593-228a18548478-metrics-tls\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.149134 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.149113    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79bf4395-e3d6-4071-aedd-548c185fa39d-crio-socket\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.149134 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.149127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79bf4395-e3d6-4071-aedd-548c185fa39d-data-volume\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.208412 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.208375    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 21:46:43.208412 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.208375    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-p5fxq\""
Oct 29 21:46:43.249942 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.249851    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmrsr\" (UniqueName: \"kubernetes.io/projected/79bf4395-e3d6-4071-aedd-548c185fa39d-kube-api-access-kmrsr\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.249942 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.249924    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3accc688-8a04-4462-9593-228a18548478-metrics-tls\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.249942 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.249951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79bf4395-e3d6-4071-aedd-548c185fa39d-crio-socket\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.250456 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.249968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79bf4395-e3d6-4071-aedd-548c185fa39d-data-volume\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.250456 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.249997    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79bf4395-e3d6-4071-aedd-548c185fa39d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.250456 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.250023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79bf4395-e3d6-4071-aedd-548c185fa39d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.250456 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.250210    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79bf4395-e3d6-4071-aedd-548c185fa39d-crio-socket\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.250456 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.250265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3accc688-8a04-4462-9593-228a18548478-config-volume\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.250456 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.250297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3accc688-8a04-4462-9593-228a18548478-tmp-dir\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.250456 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.250331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnclr\" (UniqueName: \"kubernetes.io/projected/3accc688-8a04-4462-9593-228a18548478-kube-api-access-gnclr\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.250456 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.250365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6b42\" (UniqueName: \"kubernetes.io/projected/cd3c4499-8959-4616-842b-c556e7d23137-kube-api-access-b6b42\") pod \"ingress-canary-zv5gd\" (UID: \"cd3c4499-8959-4616-842b-c556e7d23137\") " pod="openshift-ingress-canary/ingress-canary-zv5gd"
Oct 29 21:46:43.250456 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.250397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd3c4499-8959-4616-842b-c556e7d23137-cert\") pod \"ingress-canary-zv5gd\" (UID: \"cd3c4499-8959-4616-842b-c556e7d23137\") " pod="openshift-ingress-canary/ingress-canary-zv5gd"
Oct 29 21:46:43.250848 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.250497    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79bf4395-e3d6-4071-aedd-548c185fa39d-data-volume\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.250848 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.250649    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3accc688-8a04-4462-9593-228a18548478-tmp-dir\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.250848 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.250768    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79bf4395-e3d6-4071-aedd-548c185fa39d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.250938 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.250857    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3accc688-8a04-4462-9593-228a18548478-config-volume\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.254169 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.254144    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3accc688-8a04-4462-9593-228a18548478-metrics-tls\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.254239 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.254166    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79bf4395-e3d6-4071-aedd-548c185fa39d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.275222 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.275140    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnclr\" (UniqueName: \"kubernetes.io/projected/3accc688-8a04-4462-9593-228a18548478-kube-api-access-gnclr\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.281195 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.281162    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmrsr\" (UniqueName: \"kubernetes.io/projected/79bf4395-e3d6-4071-aedd-548c185fa39d-kube-api-access-kmrsr\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.306011 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.305975    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:46:43.310391 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.310360    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:43.351529 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.351484    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6b42\" (UniqueName: \"kubernetes.io/projected/cd3c4499-8959-4616-842b-c556e7d23137-kube-api-access-b6b42\") pod \"ingress-canary-zv5gd\" (UID: \"cd3c4499-8959-4616-842b-c556e7d23137\") " pod="openshift-ingress-canary/ingress-canary-zv5gd"
Oct 29 21:46:43.351813 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.351788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd3c4499-8959-4616-842b-c556e7d23137-cert\") pod \"ingress-canary-zv5gd\" (UID: \"cd3c4499-8959-4616-842b-c556e7d23137\") " pod="openshift-ingress-canary/ingress-canary-zv5gd"
Oct 29 21:46:43.354584 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.354533    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd3c4499-8959-4616-842b-c556e7d23137-cert\") pod \"ingress-canary-zv5gd\" (UID: \"cd3c4499-8959-4616-842b-c556e7d23137\") " pod="openshift-ingress-canary/ingress-canary-zv5gd"
Oct 29 21:46:43.377830 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.377795    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6b42\" (UniqueName: \"kubernetes.io/projected/cd3c4499-8959-4616-842b-c556e7d23137-kube-api-access-b6b42\") pod \"ingress-canary-zv5gd\" (UID: \"cd3c4499-8959-4616-842b-c556e7d23137\") " pod="openshift-ingress-canary/ingress-canary-zv5gd"
Oct 29 21:46:43.418015 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.417974    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zv5gd"
Oct 29 21:46:43.657025 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.656986    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2867l"]
Oct 29 21:46:43.660188 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.660161    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-245p6"]
Oct 29 21:46:43.663073 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:46:43.663046    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bf4395_e3d6_4071_aedd_548c185fa39d.slice/crio-3d812d8da771abbedc07f0f5ed688a977dbdce8dae239dad6e12b15651d5b83b WatchSource:0}: Error finding container 3d812d8da771abbedc07f0f5ed688a977dbdce8dae239dad6e12b15651d5b83b: Status 404 returned error can't find the container with id 3d812d8da771abbedc07f0f5ed688a977dbdce8dae239dad6e12b15651d5b83b
Oct 29 21:46:43.663448 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:46:43.663424    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3accc688_8a04_4462_9593_228a18548478.slice/crio-66df3707a72a218383de890083a85d74eae68aee8ee9d173562b1d62817fae0a WatchSource:0}: Error finding container 66df3707a72a218383de890083a85d74eae68aee8ee9d173562b1d62817fae0a: Status 404 returned error can't find the container with id 66df3707a72a218383de890083a85d74eae68aee8ee9d173562b1d62817fae0a
Oct 29 21:46:43.729079 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:43.729055    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zv5gd"]
Oct 29 21:46:43.731225 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:46:43.731194    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3c4499_8959_4616_842b_c556e7d23137.slice/crio-3239833dfc0a5983fdfc923651715e570c2cc3757bc1eaf4131ee67a1750d4d3 WatchSource:0}: Error finding container 3239833dfc0a5983fdfc923651715e570c2cc3757bc1eaf4131ee67a1750d4d3: Status 404 returned error can't find the container with id 3239833dfc0a5983fdfc923651715e570c2cc3757bc1eaf4131ee67a1750d4d3
Oct 29 21:46:44.146026 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.145985    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-779475d747-twkqw"]
Oct 29 21:46:44.148244 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.148217    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.164645 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.164589    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 29 21:46:44.164850 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.164744    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 29 21:46:44.164850 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.164833    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 29 21:46:44.165011 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.164974    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 29 21:46:44.165174 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.165116    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 29 21:46:44.176877 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.176847    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 29 21:46:44.177065 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.176944    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6fjlx\""
Oct 29 21:46:44.177065 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.177018    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 29 21:46:44.178088 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.178065    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-779475d747-twkqw"]
Oct 29 21:46:44.228277 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.228240    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:46:44.228476 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.228247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:46:44.228476 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.228406    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:46:44.256514 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.256481    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-oauth-config\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.256995 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.256519    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-tmp\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.256995 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.256566    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-oauth-serving-cert\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.256995 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.256603    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-serving-cert\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.256995 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.256665    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-service-ca\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.256995 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.256698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drx2d\" (UniqueName: \"kubernetes.io/projected/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-kube-api-access-drx2d\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.256995 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.256757    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-config\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.264546 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.264516    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 21:46:44.264699 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.264529    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 21:46:44.265312 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.265287    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4sldf\""
Oct 29 21:46:44.265415 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.265391    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 21:46:44.265515 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.265474    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hzxgj\""
Oct 29 21:46:44.265612 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.265596    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 21:46:44.265731 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.265699    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsffp\""
Oct 29 21:46:44.358832 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.357259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-config\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.358832 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.357322    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-oauth-config\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.358832 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.357345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-tmp\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.358832 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.357386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-oauth-serving-cert\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.358832 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.357420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-serving-cert\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.358832 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.357443    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-service-ca\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.358832 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.357469    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drx2d\" (UniqueName: \"kubernetes.io/projected/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-kube-api-access-drx2d\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.358832 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.358074    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-config\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.358832 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.358520    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-oauth-serving-cert\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.359663 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.359505    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-tmp\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.359663 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.359536    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-service-ca\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.361773 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.361746    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-oauth-config\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.363577 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.363546    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-serving-cert\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.419050 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.418966    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drx2d\" (UniqueName: \"kubernetes.io/projected/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-kube-api-access-drx2d\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.462783 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.462746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:44.498639 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.498583    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zv5gd" event={"ID":"cd3c4499-8959-4616-842b-c556e7d23137","Type":"ContainerStarted","Data":"3239833dfc0a5983fdfc923651715e570c2cc3757bc1eaf4131ee67a1750d4d3"}
Oct 29 21:46:44.500440 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.500369    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-245p6" event={"ID":"3accc688-8a04-4462-9593-228a18548478","Type":"ContainerStarted","Data":"66df3707a72a218383de890083a85d74eae68aee8ee9d173562b1d62817fae0a"}
Oct 29 21:46:44.502906 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.502873    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2867l" event={"ID":"79bf4395-e3d6-4071-aedd-548c185fa39d","Type":"ContainerStarted","Data":"18c36558489d161e9978501d2bac71239bfbe04101299791e8ac1e4b876ee4f4"}
Oct 29 21:46:44.503059 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.502940    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2867l" event={"ID":"79bf4395-e3d6-4071-aedd-548c185fa39d","Type":"ContainerStarted","Data":"19e307fa05198cdeafd20f050c13e9139d581bf459791f275db7835a46c9c3f9"}
Oct 29 21:46:44.503059 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.502967    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2867l" event={"ID":"79bf4395-e3d6-4071-aedd-548c185fa39d","Type":"ContainerStarted","Data":"3d812d8da771abbedc07f0f5ed688a977dbdce8dae239dad6e12b15651d5b83b"}
Oct 29 21:46:44.785684 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.785583    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-779475d747-twkqw"]
Oct 29 21:46:44.792232 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:46:44.792150    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f0e68b_09b9_49e8_bce4_cae4e15ec43f.slice/crio-2b7ac84a5de67fab87859f8db39452d0ad41c49da9b34418888d7ff01e658423 WatchSource:0}: Error finding container 2b7ac84a5de67fab87859f8db39452d0ad41c49da9b34418888d7ff01e658423: Status 404 returned error can't find the container with id 2b7ac84a5de67fab87859f8db39452d0ad41c49da9b34418888d7ff01e658423
Oct 29 21:46:44.833056 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.833021    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"]
Oct 29 21:46:44.835768 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.835738    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:44.857252 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.857216    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 29 21:46:44.857936 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.857913    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 29 21:46:44.858179 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.858160    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6g5r7\""
Oct 29 21:46:44.869735 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.869701    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 21:46:44.869735 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.869710    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 21:46:44.870000 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.869775    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 21:46:44.876844 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.876806    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"]
Oct 29 21:46:44.891521 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.891485    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"]
Oct 29 21:46:44.896336 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.895893    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:44.923334 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.922535    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"]
Oct 29 21:46:44.958744 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.958707    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 29 21:46:44.958931 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.958849    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 29 21:46:44.958989 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.958934    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 29 21:46:44.959054 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.959034    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zcf2k\""
Oct 29 21:46:44.962938 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.962906    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wtbq7"]
Oct 29 21:46:44.963182 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.963156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:44.963247 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.963226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:44.963385 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.963351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:44.963519 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.963402    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwrg\" (UniqueName: \"kubernetes.io/projected/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-kube-api-access-ldwrg\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:44.966235 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:44.966208    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.059851 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.059593    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 21:46:45.059851 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.059705    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 21:46:45.059851 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.059753    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pjw5g\""
Oct 29 21:46:45.059851 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.059823    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 21:46:45.064707 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.064665    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/066041b1-7eba-4371-9470-101a7bf8fcfc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.064884 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.064723    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-textfile\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.064884 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.064759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.064884 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.064786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-tls\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.064884 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.064816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7l79\" (UniqueName: \"kubernetes.io/projected/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-api-access-b7l79\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.064884 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.064835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-root\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.064884 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.064851    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066041b1-7eba-4371-9470-101a7bf8fcfc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.064884 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.064866    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-sys\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.064884 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.064880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.065200 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.064902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b64908a-fffb-4ec2-8697-51ea5db37f3b-metrics-client-ca\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.065200 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.064925    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.065200 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.064968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.065200 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.065000    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:45.065200 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.065020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkhr5\" (UniqueName: \"kubernetes.io/projected/3b64908a-fffb-4ec2-8697-51ea5db37f3b-kube-api-access-bkhr5\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.065200 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.065039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwrg\" (UniqueName: \"kubernetes.io/projected/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-kube-api-access-ldwrg\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:45.065200 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.065080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:45.065200 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.065101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:45.065200 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.065123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.065200 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.065139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-wtmp\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.066902 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.066835    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:45.068941 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.068887    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:45.073243 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.073180    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:45.091603 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.091549    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwrg\" (UniqueName: \"kubernetes.io/projected/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-kube-api-access-ldwrg\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:45.151453 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.151411    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:46:45.166444 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkhr5\" (UniqueName: \"kubernetes.io/projected/3b64908a-fffb-4ec2-8697-51ea5db37f3b-kube-api-access-bkhr5\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.166582 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.166582 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166528    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-wtmp\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.166582 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/066041b1-7eba-4371-9470-101a7bf8fcfc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.166727 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166610    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-textfile\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.166727 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.166727 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166683    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-tls\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.166727 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166712    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7l79\" (UniqueName: \"kubernetes.io/projected/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-api-access-b7l79\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.166902 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-root\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.166902 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066041b1-7eba-4371-9470-101a7bf8fcfc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.166902 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-sys\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.166902 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.166902 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166817    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-wtmp\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.166902 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b64908a-fffb-4ec2-8697-51ea5db37f3b-metrics-client-ca\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.166902 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.167242 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.166944    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.167242 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.167050    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/066041b1-7eba-4371-9470-101a7bf8fcfc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.167368 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.167344    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-textfile\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.167590 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.167568    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.167699 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.167609    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.167699 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.167655    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-sys\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.167800 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.167693    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066041b1-7eba-4371-9470-101a7bf8fcfc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.168187 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.168022    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b64908a-fffb-4ec2-8697-51ea5db37f3b-metrics-client-ca\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.168187 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.168025    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-root\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.170696 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.170604    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-tls\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.175479 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.175414    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.177076 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.177013    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.182266 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.182207    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.192207 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.192169    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkhr5\" (UniqueName: \"kubernetes.io/projected/3b64908a-fffb-4ec2-8697-51ea5db37f3b-kube-api-access-bkhr5\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.194230 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.194204    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7l79\" (UniqueName: \"kubernetes.io/projected/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-api-access-b7l79\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.216066 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.216015    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:46:45.286291 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.285964    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:46:45.491031 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.490990    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"]
Oct 29 21:46:45.509491 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.509398    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-779475d747-twkqw" event={"ID":"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f","Type":"ContainerStarted","Data":"2b7ac84a5de67fab87859f8db39452d0ad41c49da9b34418888d7ff01e658423"}
Oct 29 21:46:45.511172 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.511093    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wtbq7" event={"ID":"3b64908a-fffb-4ec2-8697-51ea5db37f3b","Type":"ContainerStarted","Data":"6c733459254465418be5838dcabb8ad525786cf2728369479663d2335fd3dee3"}
Oct 29 21:46:45.512988 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.512886    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b" event={"ID":"8c893b43-cd05-45a0-9f7b-a2b5d72809c7","Type":"ContainerStarted","Data":"4b7b55989043bc79ff2d50ed75a4b5de802ae3abe4a56324e8389bd9efd54e44"}
Oct 29 21:46:45.572826 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:45.572730    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"]
Oct 29 21:46:45.580526 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:46:45.580487    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066041b1_7eba_4371_9470_101a7bf8fcfc.slice/crio-9999ce6f5d6e9d98d8a8f249aa37b8b9da08fa960a1f6bcf5c7b41c34d33db18 WatchSource:0}: Error finding container 9999ce6f5d6e9d98d8a8f249aa37b8b9da08fa960a1f6bcf5c7b41c34d33db18: Status 404 returned error can't find the container with id 9999ce6f5d6e9d98d8a8f249aa37b8b9da08fa960a1f6bcf5c7b41c34d33db18
Oct 29 21:46:46.285158 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.285113    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 21:46:46.290438 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.289126    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.309594 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.309545    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 29 21:46:46.309594 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.309567    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 29 21:46:46.309834 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.309672    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 29 21:46:46.317858 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.317595    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 29 21:46:46.317858 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.317677    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 29 21:46:46.317858 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.317817    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 29 21:46:46.332813 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.332776    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 21:46:46.336474 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.336076    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 29 21:46:46.379169 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.379127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.379352 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.379181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.379352 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.379208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.379352 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.379317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-out\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.379485 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.379420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-volume\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.379485 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.379458    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.379550 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.379490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.379550 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.379525    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.379616 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.379586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.379673 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.379616    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klh9r\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-kube-api-access-klh9r\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.379715 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.379679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.379754 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.379738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-web-config\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.379789 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.379767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.412188 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.412102    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 29 21:46:46.412188 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.412103    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 29 21:46:46.412188 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.412168    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9qchq\""
Oct 29 21:46:46.480414 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.480373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.480582 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.480420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klh9r\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-kube-api-access-klh9r\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.480582 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.480469    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.480582 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.480516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-web-config\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.480582 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.480547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.480724 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.480582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.480724 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.480612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.480724 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.480657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.480724 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.480690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-out\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.480724 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.480718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-volume\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.480861 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.480749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.480861 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.480778    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.480861 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.480810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.482518 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.481789    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.490006 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.489283    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.490006 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.489946    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.494217 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.494003    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-web-config\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.494856 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.494787    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-out\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.495135 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.495109    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-volume\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.495514 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.495461    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.495514 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.495473    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.497410 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.497379    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.499310 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.499224    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.502214 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.502133    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.504761 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.504581    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.519739 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.519413    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-245p6" event={"ID":"3accc688-8a04-4462-9593-228a18548478","Type":"ContainerStarted","Data":"5dfcbaf36269d5251bce52c453989b0cd0acebfaad479783416382f1e3f7a23c"}
Oct 29 21:46:46.519739 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.519524    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-245p6" event={"ID":"3accc688-8a04-4462-9593-228a18548478","Type":"ContainerStarted","Data":"e9c2fa7defb0fddd7d22e676ceb24236d430b5c8fe790922f67d5092b865fc3e"}
Oct 29 21:46:46.520088 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.520064    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:46.522836 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.522793    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b" event={"ID":"8c893b43-cd05-45a0-9f7b-a2b5d72809c7","Type":"ContainerStarted","Data":"076dde63780ca5ae89371a37a8fde8953395a3b4a72c3e223eb4ef419d2503b3"}
Oct 29 21:46:46.522985 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.522870    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b" event={"ID":"8c893b43-cd05-45a0-9f7b-a2b5d72809c7","Type":"ContainerStarted","Data":"44caaf91ead9a07d1ea745c4aa3c3d576087d76ec86e5982079a8efcc2294bda"}
Oct 29 21:46:46.524007 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.523947    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klh9r\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-kube-api-access-klh9r\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.526921 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.526883    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4" event={"ID":"066041b1-7eba-4371-9470-101a7bf8fcfc","Type":"ContainerStarted","Data":"9999ce6f5d6e9d98d8a8f249aa37b8b9da08fa960a1f6bcf5c7b41c34d33db18"}
Oct 29 21:46:46.626121 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.625794    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:46:46.976396 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.976333    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-245p6" podStartSLOduration=2.7543080460000002 podStartE2EDuration="4.976315162s" totalImagesPullingTime="2.222007116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:46.568496985 +0000 UTC m=+50.878171324" watchObservedRunningTime="2026-10-29 21:46:46.976315162 +0000 UTC m=+51.285989489"
Oct 29 21:46:46.977299 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:46.977259    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 21:46:46.982216 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:46:46.982178    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d414df_8bf2_4c06_b8f0_ef1b3ce29e57.slice/crio-bed90a2833cf42ce4a410e6c930d6ef7667ac810586d5f076818e1b5ecaad60f WatchSource:0}: Error finding container bed90a2833cf42ce4a410e6c930d6ef7667ac810586d5f076818e1b5ecaad60f: Status 404 returned error can't find the container with id bed90a2833cf42ce4a410e6c930d6ef7667ac810586d5f076818e1b5ecaad60f
Oct 29 21:46:47.018925 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.018889    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"]
Oct 29 21:46:47.025293 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.023383    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.115894 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.115829    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 29 21:46:47.115894 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.115838    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-wqq76\""
Oct 29 21:46:47.115894 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.115878    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 29 21:46:47.115894 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.115883    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 29 21:46:47.115894 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.115893    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 29 21:46:47.116301 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.115942    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9d1tb9gv0p1rk\""
Oct 29 21:46:47.116301 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.116012    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 29 21:46:47.124224 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.124183    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"]
Oct 29 21:46:47.188216 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.188170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-tls\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.188418 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.188225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8bm\" (UniqueName: \"kubernetes.io/projected/b96d72b2-bc45-4962-b529-f08155ad0215-kube-api-access-vv8bm\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.188418 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.188345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.188418 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.188413    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-grpc-tls\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.188594 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.188471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b96d72b2-bc45-4962-b529-f08155ad0215-tmp\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.188594 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.188508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.188594 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.188538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b96d72b2-bc45-4962-b529-f08155ad0215-metrics-client-ca\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.188594 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.188584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.188822 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.188677    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.289922 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.289085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-tls\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.289922 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.289162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8bm\" (UniqueName: \"kubernetes.io/projected/b96d72b2-bc45-4962-b529-f08155ad0215-kube-api-access-vv8bm\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.289922 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.289211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.289922 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.289245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-grpc-tls\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.289922 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.289305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b96d72b2-bc45-4962-b529-f08155ad0215-tmp\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.289922 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.289333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.289922 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.289357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b96d72b2-bc45-4962-b529-f08155ad0215-metrics-client-ca\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.289922 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.289405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.289922 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.289463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.290946 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.290919    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b96d72b2-bc45-4962-b529-f08155ad0215-tmp\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.291357 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.291031    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b96d72b2-bc45-4962-b529-f08155ad0215-metrics-client-ca\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.295476 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.295165    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.295476 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.295441    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.295590 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.295561    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.298082 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.298044    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-grpc-tls\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.299646 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.299599    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-tls\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.307721 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.307645    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.327332 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.327260    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8bm\" (UniqueName: \"kubernetes.io/projected/b96d72b2-bc45-4962-b529-f08155ad0215-kube-api-access-vv8bm\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.357791 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.357750    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:47.534464 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.534428    2582 generic.go:356] "Generic (PLEG): container finished" podID="3b64908a-fffb-4ec2-8697-51ea5db37f3b" containerID="8c0116cd936f5877e4cb58a85f0cf1a63a8423bc9fcb1aac5815e01232ac54d8" exitCode=0
Oct 29 21:46:47.534835 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.534805    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wtbq7" event={"ID":"3b64908a-fffb-4ec2-8697-51ea5db37f3b","Type":"ContainerDied","Data":"8c0116cd936f5877e4cb58a85f0cf1a63a8423bc9fcb1aac5815e01232ac54d8"}
Oct 29 21:46:47.540123 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.540090    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b" event={"ID":"8c893b43-cd05-45a0-9f7b-a2b5d72809c7","Type":"ContainerStarted","Data":"40b4adce4a420bb584b48258790d6a4ac2383f1935883868b7ed6e24335880c9"}
Oct 29 21:46:47.543946 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.543905    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2867l" event={"ID":"79bf4395-e3d6-4071-aedd-548c185fa39d","Type":"ContainerStarted","Data":"2fb409a12e938fe252ac56bcfb0a6928917d1c4dcc10d5d3774fa1d900760984"}
Oct 29 21:46:47.545698 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.545665    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerStarted","Data":"bed90a2833cf42ce4a410e6c930d6ef7667ac810586d5f076818e1b5ecaad60f"}
Oct 29 21:46:47.681533 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.681467    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2867l" podStartSLOduration=2.630609477 podStartE2EDuration="5.68144683s" totalImagesPullingTime="3.050837353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:46:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:47.634119229 +0000 UTC m=+51.943793555" watchObservedRunningTime="2026-10-29 21:46:47.68144683 +0000 UTC m=+51.991121157"
Oct 29 21:46:47.682668 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.682517    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b" podStartSLOduration=2.10791077 podStartE2EDuration="3.682503912s" totalImagesPullingTime="1.574593142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:47.681144559 +0000 UTC m=+51.990818888" watchObservedRunningTime="2026-10-29 21:46:47.682503912 +0000 UTC m=+51.992178238"
Oct 29 21:46:47.709165 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:47.708798    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"]
Oct 29 21:46:47.710019 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:46:47.709959    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96d72b2_bc45_4962_b529_f08155ad0215.slice/crio-32380f5c9fc4b9aa3b0dfab906c26f7d8179c9ff63f0b35e0533089efb12c27b WatchSource:0}: Error finding container 32380f5c9fc4b9aa3b0dfab906c26f7d8179c9ff63f0b35e0533089efb12c27b: Status 404 returned error can't find the container with id 32380f5c9fc4b9aa3b0dfab906c26f7d8179c9ff63f0b35e0533089efb12c27b
Oct 29 21:46:48.553363 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:48.553329    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wtbq7" event={"ID":"3b64908a-fffb-4ec2-8697-51ea5db37f3b","Type":"ContainerStarted","Data":"e094d7725832dba4634bb7d4fb38ea69bf1f01144fb34586add4960c3e3a1b7b"}
Oct 29 21:46:48.553874 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:48.553394    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wtbq7" event={"ID":"3b64908a-fffb-4ec2-8697-51ea5db37f3b","Type":"ContainerStarted","Data":"ec61e553a1852b48f92fec28e651546f120514b98c52114eb5e3895e75fd837f"}
Oct 29 21:46:48.555304 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:48.555266    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb" event={"ID":"b96d72b2-bc45-4962-b529-f08155ad0215","Type":"ContainerStarted","Data":"32380f5c9fc4b9aa3b0dfab906c26f7d8179c9ff63f0b35e0533089efb12c27b"}
Oct 29 21:46:48.557517 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:48.557484    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zv5gd" event={"ID":"cd3c4499-8959-4616-842b-c556e7d23137","Type":"ContainerStarted","Data":"88c42700a3f1bcd1a5b78323300f19ab0cf8b9604e3d18ad17a0478b77b8fab0"}
Oct 29 21:46:48.560682 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:48.560642    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4" event={"ID":"066041b1-7eba-4371-9470-101a7bf8fcfc","Type":"ContainerStarted","Data":"1ae5f6e6d0b30e80b455e784df9a0f7209c6f4b4b370cc25b276cbc6a2b18df8"}
Oct 29 21:46:48.560837 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:48.560715    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4" event={"ID":"066041b1-7eba-4371-9470-101a7bf8fcfc","Type":"ContainerStarted","Data":"4f679c24179724a04dc6f5e64768fb0ee4f37fbc3306758828a3d9600a0891ea"}
Oct 29 21:46:48.560837 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:48.560743    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4" event={"ID":"066041b1-7eba-4371-9470-101a7bf8fcfc","Type":"ContainerStarted","Data":"9af8b5b8b1d5723562af40db5cbb29cf122ea28e9b99563f68cbe3787da9524c"}
Oct 29 21:46:48.607355 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:48.607289    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wtbq7" podStartSLOduration=2.904009382 podStartE2EDuration="4.607269777s" totalImagesPullingTime="1.575569944s" totalInitContainerRuntime="127.690451ms" isStatefulPod=true podCreationTimestamp="2026-10-29 21:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:48.606708302 +0000 UTC m=+52.916382628" watchObservedRunningTime="2026-10-29 21:46:48.607269777 +0000 UTC m=+52.916944104"
Oct 29 21:46:48.711647 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:48.711576    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zv5gd" podStartSLOduration=1.762323602 podStartE2EDuration="5.711559626s" totalImagesPullingTime="3.949236024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:48.711231766 +0000 UTC m=+53.020906106" watchObservedRunningTime="2026-10-29 21:46:48.711559626 +0000 UTC m=+53.021233953"
Oct 29 21:46:48.763235 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:48.763126    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4" podStartSLOduration=2.791465468 podStartE2EDuration="4.763111465s" totalImagesPullingTime="1.971645997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:48.762734632 +0000 UTC m=+53.072408958" watchObservedRunningTime="2026-10-29 21:46:48.763111465 +0000 UTC m=+53.072785848"
Oct 29 21:46:49.530470 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.530376    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"]
Oct 29 21:46:49.534261 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.534229    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:46:49.550291 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.550259    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f7fc46b4-vvxlv"]
Oct 29 21:46:49.553257 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.553228    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lssrn\""
Oct 29 21:46:49.553698 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.553681    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.561790 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.561756    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 29 21:46:49.563999 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.563855    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"]
Oct 29 21:46:49.569710 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.569674    2582 generic.go:356] "Generic (PLEG): container finished" podID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerID="853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26" exitCode=0
Oct 29 21:46:49.569883 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.569787    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerDied","Data":"853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26"}
Oct 29 21:46:49.573819 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.573778    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-779475d747-twkqw" event={"ID":"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f","Type":"ContainerStarted","Data":"7bd4054d7b4256f685efd5ef8fedbb8141a441169fcb2bdaaa771d8911af1108"}
Oct 29 21:46:49.577719 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.577431    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f7fc46b4-vvxlv"]
Oct 29 21:46:49.582596 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.582565    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 29 21:46:49.582812 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.582680    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 29 21:46:49.624544 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.624491    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-779475d747-twkqw" podStartSLOduration=1.207035508 podStartE2EDuration="5.624475876s" totalImagesPullingTime="4.417440368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:49.624168646 +0000 UTC m=+53.933842972" watchObservedRunningTime="2026-10-29 21:46:49.624475876 +0000 UTC m=+53.934150203"
Oct 29 21:46:49.654398 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.654358    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 29 21:46:49.654590 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.654507    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 29 21:46:49.654709 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.654616    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fzqww\""
Oct 29 21:46:49.654816 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.654795    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-296ss7dcv4g3a\""
Oct 29 21:46:49.715073 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.715034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-client-ca-bundle\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.715273 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.715099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/baaab207-5534-4ef9-ac3a-9466ea5b900e-audit-log\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.715413 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.715383    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5fvk\" (UniqueName: \"kubernetes.io/projected/4c2dd590-c639-42fe-8a02-4945b6403b3b-kube-api-access-t5fvk\") pod \"monitoring-plugin-77f4f76594-wc4l2\" (UID: \"4c2dd590-c639-42fe-8a02-4945b6403b3b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:46:49.715547 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.715486    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-secret-metrics-server-tls\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.715722 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.715698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkc5w\" (UniqueName: \"kubernetes.io/projected/baaab207-5534-4ef9-ac3a-9466ea5b900e-kube-api-access-hkc5w\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.715791 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.715753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c2dd590-c639-42fe-8a02-4945b6403b3b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wc4l2\" (UID: \"4c2dd590-c639-42fe-8a02-4945b6403b3b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:46:49.715791 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.715784    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-secret-metrics-server-client-certs\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.716588 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.716550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baaab207-5534-4ef9-ac3a-9466ea5b900e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.716879 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.716854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/baaab207-5534-4ef9-ac3a-9466ea5b900e-metrics-server-audit-profiles\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.818052 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.818014    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-secret-metrics-server-tls\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.818256 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.818092    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkc5w\" (UniqueName: \"kubernetes.io/projected/baaab207-5534-4ef9-ac3a-9466ea5b900e-kube-api-access-hkc5w\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.818256 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.818126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c2dd590-c639-42fe-8a02-4945b6403b3b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wc4l2\" (UID: \"4c2dd590-c639-42fe-8a02-4945b6403b3b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:46:49.818256 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.818146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-secret-metrics-server-client-certs\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.818256 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.818193    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baaab207-5534-4ef9-ac3a-9466ea5b900e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.818256 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.818220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/baaab207-5534-4ef9-ac3a-9466ea5b900e-metrics-server-audit-profiles\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.818256 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.818255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-client-ca-bundle\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.818443 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.818275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/baaab207-5534-4ef9-ac3a-9466ea5b900e-audit-log\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.818443 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.818294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5fvk\" (UniqueName: \"kubernetes.io/projected/4c2dd590-c639-42fe-8a02-4945b6403b3b-kube-api-access-t5fvk\") pod \"monitoring-plugin-77f4f76594-wc4l2\" (UID: \"4c2dd590-c639-42fe-8a02-4945b6403b3b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:46:49.818986 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.818946    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/baaab207-5534-4ef9-ac3a-9466ea5b900e-audit-log\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.819265 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.819204    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baaab207-5534-4ef9-ac3a-9466ea5b900e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.819859 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.819744    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/baaab207-5534-4ef9-ac3a-9466ea5b900e-metrics-server-audit-profiles\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.821530 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.821505    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-secret-metrics-server-tls\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.821671 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.821509    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c2dd590-c639-42fe-8a02-4945b6403b3b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wc4l2\" (UID: \"4c2dd590-c639-42fe-8a02-4945b6403b3b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:46:49.821735 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.821688    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-secret-metrics-server-client-certs\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.822136 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.822119    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-client-ca-bundle\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.843759 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.843720    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5fvk\" (UniqueName: \"kubernetes.io/projected/4c2dd590-c639-42fe-8a02-4945b6403b3b-kube-api-access-t5fvk\") pod \"monitoring-plugin-77f4f76594-wc4l2\" (UID: \"4c2dd590-c639-42fe-8a02-4945b6403b3b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:46:49.843929 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.843729    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkc5w\" (UniqueName: \"kubernetes.io/projected/baaab207-5534-4ef9-ac3a-9466ea5b900e-kube-api-access-hkc5w\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:49.849397 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.849366    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:46:49.869164 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:49.869129    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:46:50.148761 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.148719    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-54969c98b4-7zdlx"]
Oct 29 21:46:50.153049 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.153019    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.159859 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.159467    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"]
Oct 29 21:46:50.175815 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.174791    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 29 21:46:50.175815 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.175389    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 29 21:46:50.175815 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.175473    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 29 21:46:50.175815 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.175575    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ghrw4\""
Oct 29 21:46:50.175815 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.175661    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 29 21:46:50.175815 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.175768    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 29 21:46:50.188352 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.188316    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 29 21:46:50.190592 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.190474    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54969c98b4-7zdlx"]
Oct 29 21:46:50.195904 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.195872    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f7fc46b4-vvxlv"]
Oct 29 21:46:50.200198 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:46:50.200158    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaaab207_5534_4ef9_ac3a_9466ea5b900e.slice/crio-5b96159b5793d1be00bf8b618ba9cdf13625516b0c1b39e7ccea9a4794d29467 WatchSource:0}: Error finding container 5b96159b5793d1be00bf8b618ba9cdf13625516b0c1b39e7ccea9a4794d29467: Status 404 returned error can't find the container with id 5b96159b5793d1be00bf8b618ba9cdf13625516b0c1b39e7ccea9a4794d29467
Oct 29 21:46:50.323075 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.323031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-secret-telemeter-client\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.323269 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.323184    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-serving-certs-ca-bundle\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.323269 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.323217    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-metrics-client-ca\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.323269 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.323254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9d2j\" (UniqueName: \"kubernetes.io/projected/dab249e6-5406-4ca6-87d5-d6fe41de7754-kube-api-access-t9d2j\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.323415 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.323368    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-telemeter-client-tls\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.323415 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.323402    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-federate-client-tls\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.323564 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.323447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.323564 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.323499    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.424682 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.424576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-secret-telemeter-client\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.424841 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.424708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-serving-certs-ca-bundle\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.424841 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.424747    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-metrics-client-ca\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.424948 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.424916    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9d2j\" (UniqueName: \"kubernetes.io/projected/dab249e6-5406-4ca6-87d5-d6fe41de7754-kube-api-access-t9d2j\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.425067 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.424971    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-telemeter-client-tls\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.425067 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.424996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-federate-client-tls\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.425067 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.425040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.425221 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.425067    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.425557 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.425531    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-metrics-client-ca\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.425690 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.425572    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-serving-certs-ca-bundle\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.426190 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.426144    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.427949 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.427820    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-secret-telemeter-client\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.428896 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.428835    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-telemeter-client-tls\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.428896 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.428842    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-federate-client-tls\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.428896 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.428869    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.451137 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.451103    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9d2j\" (UniqueName: \"kubernetes.io/projected/dab249e6-5406-4ca6-87d5-d6fe41de7754-kube-api-access-t9d2j\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.473410 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.473371    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:46:50.579718 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.579677    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb" event={"ID":"b96d72b2-bc45-4962-b529-f08155ad0215","Type":"ContainerStarted","Data":"d2382126ba442c0efe22ff3620c32d0b7f066f2728f33babc6901918c012f6d8"}
Oct 29 21:46:50.580117 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.579774    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb" event={"ID":"b96d72b2-bc45-4962-b529-f08155ad0215","Type":"ContainerStarted","Data":"ff91a0be2a603d544474712cc211367644460ab893d500d3b4ea347258cb6175"}
Oct 29 21:46:50.580117 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.579817    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb" event={"ID":"b96d72b2-bc45-4962-b529-f08155ad0215","Type":"ContainerStarted","Data":"e3f33e7e3973af37ccd391e9c780a50cb1eb1d9db3bddd140158717cd9f61fd8"}
Oct 29 21:46:50.581007 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.580969    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv" event={"ID":"baaab207-5534-4ef9-ac3a-9466ea5b900e","Type":"ContainerStarted","Data":"5b96159b5793d1be00bf8b618ba9cdf13625516b0c1b39e7ccea9a4794d29467"}
Oct 29 21:46:50.582378 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.582346    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2" event={"ID":"4c2dd590-c639-42fe-8a02-4945b6403b3b","Type":"ContainerStarted","Data":"1b5c45168936ec68706fa22cf5010febbd2abb5b13240dcbafa697326f78db6f"}
Oct 29 21:46:50.827686 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:50.827645    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54969c98b4-7zdlx"]
Oct 29 21:46:50.847013 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:46:50.846978    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab249e6_5406_4ca6_87d5_d6fe41de7754.slice/crio-3259f5e5de34085e454c4fec63e4c26517488e625b618e564cdc7fc4f3ffe1d9 WatchSource:0}: Error finding container 3259f5e5de34085e454c4fec63e4c26517488e625b618e564cdc7fc4f3ffe1d9: Status 404 returned error can't find the container with id 3259f5e5de34085e454c4fec63e4c26517488e625b618e564cdc7fc4f3ffe1d9
Oct 29 21:46:51.465781 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:51.464806    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:46:51.589057 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:51.589015    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx" event={"ID":"dab249e6-5406-4ca6-87d5-d6fe41de7754","Type":"ContainerStarted","Data":"3259f5e5de34085e454c4fec63e4c26517488e625b618e564cdc7fc4f3ffe1d9"}
Oct 29 21:46:51.593452 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:51.593419    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb" event={"ID":"b96d72b2-bc45-4962-b529-f08155ad0215","Type":"ContainerStarted","Data":"0a4ecce09f9d4ee5a6f61ad7b297bacd0fbf07200f32505b5f59606b2155e719"}
Oct 29 21:46:51.593660 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:51.593488    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb" event={"ID":"b96d72b2-bc45-4962-b529-f08155ad0215","Type":"ContainerStarted","Data":"519921b28669016403b22b242b77a584ae10600ad9be78755648b5129217b923"}
Oct 29 21:46:51.593660 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:51.593512    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb" event={"ID":"b96d72b2-bc45-4962-b529-f08155ad0215","Type":"ContainerStarted","Data":"3112c5ed1a4856d04a0016444bd57254e0cd87aca583d46040378bb76eefe419"}
Oct 29 21:46:51.593660 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:51.593559    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:46:51.598246 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:51.598074    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerStarted","Data":"94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28"}
Oct 29 21:46:51.598246 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:51.598167    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerStarted","Data":"20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b"}
Oct 29 21:46:51.647123 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:51.647046    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb" podStartSLOduration=2.4470093520000002 podStartE2EDuration="5.647024925s" totalImagesPullingTime="3.200015573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:51.645477815 +0000 UTC m=+55.955152144" watchObservedRunningTime="2026-10-29 21:46:51.647024925 +0000 UTC m=+55.956699254"
Oct 29 21:46:52.607508 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:52.607409    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv" event={"ID":"baaab207-5534-4ef9-ac3a-9466ea5b900e","Type":"ContainerStarted","Data":"52c4084d716c9dfc8cc172fca4fa969be32130c07c7c9b3f89a5fe4e8176a73e"}
Oct 29 21:46:52.611573 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:52.611535    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerStarted","Data":"9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e"}
Oct 29 21:46:52.611788 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:52.611616    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerStarted","Data":"954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99"}
Oct 29 21:46:52.611788 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:52.611662    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerStarted","Data":"c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2"}
Oct 29 21:46:52.611788 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:52.611689    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerStarted","Data":"4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d"}
Oct 29 21:46:52.613290 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:52.613253    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2" event={"ID":"4c2dd590-c639-42fe-8a02-4945b6403b3b","Type":"ContainerStarted","Data":"3ba7fe7674dd613c6b429f4eb4bee70d0a9cde57ee0e7b16b244ea5846791cd9"}
Oct 29 21:46:52.613723 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:52.613671    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:46:52.618918 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:52.618888    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:46:52.655688 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:52.655615    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv" podStartSLOduration=1.717460711 podStartE2EDuration="3.655599301s" totalImagesPullingTime="1.93813859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:52.65486548 +0000 UTC m=+56.964539806" watchObservedRunningTime="2026-10-29 21:46:52.655599301 +0000 UTC m=+56.965273627"
Oct 29 21:46:52.719392 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:52.719336    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.476276543 podStartE2EDuration="6.719320382s" totalImagesPullingTime="3.22204444s" totalInitContainerRuntime="20.999399ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:52.712883168 +0000 UTC m=+57.022557533" watchObservedRunningTime="2026-10-29 21:46:52.719320382 +0000 UTC m=+57.028994708"
Oct 29 21:46:53.175888 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.175785    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2" podStartSLOduration=2.318074213 podStartE2EDuration="4.175766813s" totalImagesPullingTime="1.8576926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:52.759439691 +0000 UTC m=+57.069114017" watchObservedRunningTime="2026-10-29 21:46:53.175766813 +0000 UTC m=+57.485441139"
Oct 29 21:46:53.176277 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.176262    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c7d74bc46-5bb2n"]
Oct 29 21:46:53.179610 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.179591    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.211030 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.211000    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 29 21:46:53.212315 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.212289    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7d74bc46-5bb2n"]
Oct 29 21:46:53.265414 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.265366    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-tmp\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.265414 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.265414    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-config\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.265654 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.265523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-oauth-config\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.265654 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.265555    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-service-ca\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.265654 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.265573    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-serving-cert\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.265759 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.265676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-trusted-ca-bundle\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.265794 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.265750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcnrm\" (UniqueName: \"kubernetes.io/projected/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-kube-api-access-vcnrm\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.265823 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.265798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-oauth-serving-cert\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.366960 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.366919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-tmp\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.367127 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.366974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-config\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.367127 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.367029    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-oauth-config\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.367127 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.367076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-service-ca\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.367127 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.367104    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-serving-cert\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.367280 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.367143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-trusted-ca-bundle\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.367280 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.367176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcnrm\" (UniqueName: \"kubernetes.io/projected/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-kube-api-access-vcnrm\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.367280 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.367225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-oauth-serving-cert\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.367404 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.367384    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-tmp\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.368100 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.368070    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-service-ca\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.368199 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.368071    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-oauth-serving-cert\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.368334 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.368310    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-config\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.368334 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.368318    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-trusted-ca-bundle\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.370956 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.370923    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-oauth-config\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.371185 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.371160    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-serving-cert\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.393806 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.393772    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcnrm\" (UniqueName: \"kubernetes.io/projected/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-kube-api-access-vcnrm\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.491991 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.491889    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:46:53.619070 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.619035    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx" event={"ID":"dab249e6-5406-4ca6-87d5-d6fe41de7754","Type":"ContainerStarted","Data":"39e4b8d656a6794e2490e402ee7a76ad40d31a3d0b29018456ecda56579493b9"}
Oct 29 21:46:53.619555 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.619101    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx" event={"ID":"dab249e6-5406-4ca6-87d5-d6fe41de7754","Type":"ContainerStarted","Data":"52c91b76bcc66773024b61de09ac5bf7b8af11f7b9a24c05ad131d1f2a8d3f25"}
Oct 29 21:46:53.619555 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.619128    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx" event={"ID":"dab249e6-5406-4ca6-87d5-d6fe41de7754","Type":"ContainerStarted","Data":"41a8b4a21a3c17000c0c878d9b90905ff9ff145bb34cc9def59747edb1996d5e"}
Oct 29 21:46:53.671282 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.671229    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx" podStartSLOduration=1.7458625319999999 podStartE2EDuration="3.671213093s" totalImagesPullingTime="1.925350561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:53.670101193 +0000 UTC m=+57.979775521" watchObservedRunningTime="2026-10-29 21:46:53.671213093 +0000 UTC m=+57.980887419"
Oct 29 21:46:53.799793 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:53.799765    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7d74bc46-5bb2n"]
Oct 29 21:46:53.802302 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:46:53.802272    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20e4d88_bb47_4294_8242_2e71ca6d3f6c.slice/crio-dbf589c6e0dabaec0d066764594086a44f6bc30d6f4d74670ad6dca76ed2ad62 WatchSource:0}: Error finding container dbf589c6e0dabaec0d066764594086a44f6bc30d6f4d74670ad6dca76ed2ad62: Status 404 returned error can't find the container with id dbf589c6e0dabaec0d066764594086a44f6bc30d6f4d74670ad6dca76ed2ad62
Oct 29 21:46:54.463855 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:54.463797    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:54.464064 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:54.463877    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:54.468740 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:54.468716    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:54.625374 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:54.625334    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7d74bc46-5bb2n" event={"ID":"c20e4d88-bb47-4294-8242-2e71ca6d3f6c","Type":"ContainerStarted","Data":"3e24598f21593450c2d74dd245c7e519161ac49e41a357f860e17fc275e3dead"}
Oct 29 21:46:54.625849 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:54.625399    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7d74bc46-5bb2n" event={"ID":"c20e4d88-bb47-4294-8242-2e71ca6d3f6c","Type":"ContainerStarted","Data":"dbf589c6e0dabaec0d066764594086a44f6bc30d6f4d74670ad6dca76ed2ad62"}
Oct 29 21:46:54.629736 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:54.629708    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:46:54.672465 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:54.672408    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c7d74bc46-5bb2n" podStartSLOduration=1.672390456 podStartE2EDuration="1.672390456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:54.67161233 +0000 UTC m=+58.981286658" watchObservedRunningTime="2026-10-29 21:46:54.672390456 +0000 UTC m=+58.982064782"
Oct 29 21:46:57.565586 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:57.565556    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-245p6"
Oct 29 21:46:57.621087 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:46:57.621059    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:00.382827 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.382791    2582 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7d74bc46-5bb2n"]
Oct 29 21:47:00.419441 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.419397    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-968f5bf44-dhgp8"]
Oct 29 21:47:00.424482 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.424449    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.453794 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.453753    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-968f5bf44-dhgp8"]
Oct 29 21:47:00.552699 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.552657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz6vz\" (UniqueName: \"kubernetes.io/projected/2f64c76d-6e73-4cc0-9043-8f6235e0e281-kube-api-access-sz6vz\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.552894 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.552720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-serving-cert\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.552894 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.552743    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f64c76d-6e73-4cc0-9043-8f6235e0e281-tmp\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.552894 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.552791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-oauth-config\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.552894 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.552815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-config\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.553055 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.552907    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-oauth-serving-cert\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.553055 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.552959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-trusted-ca-bundle\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.553126 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.553105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-service-ca\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.654478 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.654383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-service-ca\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.654478 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.654429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz6vz\" (UniqueName: \"kubernetes.io/projected/2f64c76d-6e73-4cc0-9043-8f6235e0e281-kube-api-access-sz6vz\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.654478 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.654463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-serving-cert\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.654783 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.654490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f64c76d-6e73-4cc0-9043-8f6235e0e281-tmp\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.654783 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.654662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-oauth-config\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.654783 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.654700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-config\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.654783 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.654757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-oauth-serving-cert\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.654977 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.654797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-trusted-ca-bundle\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.655033 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.654988    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f64c76d-6e73-4cc0-9043-8f6235e0e281-tmp\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.655343 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.655292    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-service-ca\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.655513 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.655495    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-oauth-serving-cert\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.655680 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.655659    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-config\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.655776 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.655754    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-trusted-ca-bundle\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.658001 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.657978    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-serving-cert\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.658104 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.657982    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-oauth-config\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.686529 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.686493    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz6vz\" (UniqueName: \"kubernetes.io/projected/2f64c76d-6e73-4cc0-9043-8f6235e0e281-kube-api-access-sz6vz\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:00.738984 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:00.738945    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:01.079659 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:01.078480    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-968f5bf44-dhgp8"]
Oct 29 21:47:01.083705 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:47:01.083673    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f64c76d_6e73_4cc0_9043_8f6235e0e281.slice/crio-60374fa76ee4647cf89872d467fec40c6a20477a19eccc4a213e21518f86c57a WatchSource:0}: Error finding container 60374fa76ee4647cf89872d467fec40c6a20477a19eccc4a213e21518f86c57a: Status 404 returned error can't find the container with id 60374fa76ee4647cf89872d467fec40c6a20477a19eccc4a213e21518f86c57a
Oct 29 21:47:01.657383 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:01.657344    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-968f5bf44-dhgp8" event={"ID":"2f64c76d-6e73-4cc0-9043-8f6235e0e281","Type":"ContainerStarted","Data":"30669bfe69d6bd3c22b1dc115c5f2eec23d794755b761619f187baace29b5951"}
Oct 29 21:47:01.657917 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:01.657394    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-968f5bf44-dhgp8" event={"ID":"2f64c76d-6e73-4cc0-9043-8f6235e0e281","Type":"ContainerStarted","Data":"60374fa76ee4647cf89872d467fec40c6a20477a19eccc4a213e21518f86c57a"}
Oct 29 21:47:01.703825 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:01.703764    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-968f5bf44-dhgp8" podStartSLOduration=1.703749864 podStartE2EDuration="1.703749864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:01.70367405 +0000 UTC m=+66.013348376" watchObservedRunningTime="2026-10-29 21:47:01.703749864 +0000 UTC m=+66.013424189"
Oct 29 21:47:01.972812 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:01.972719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:47:01.972812 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:01.972795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:47:01.987362 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:01.987335    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 21:47:01.987444 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:01.987384    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 21:47:01.996061 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:01.996019    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:47:01.996195 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:01.996116    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:47:02.073820 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.073777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht42j\" (UniqueName: \"kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j\") pod \"network-check-target-n8m99\" (UID: \"0fadb982-a597-4489-ae16-3dc4fb5a7810\") " pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:47:02.088298 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.088265    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 21:47:02.108798 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.108762    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 21:47:02.117575 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.117540    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht42j\" (UniqueName: \"kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j\") pod \"network-check-target-n8m99\" (UID: \"0fadb982-a597-4489-ae16-3dc4fb5a7810\") " pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:47:02.258601 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.258511    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4sldf\""
Oct 29 21:47:02.264797 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.264770    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:47:02.271563 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.271529    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsffp\""
Oct 29 21:47:02.277850 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.277823    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:47:02.299597 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.299570    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hzxgj\""
Oct 29 21:47:02.305732 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.305701    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:47:02.575111 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.575086    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vth7d"]
Oct 29 21:47:02.577955 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:47:02.577922    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae94d17c_b881_46e1_8063_e88c9dc53369.slice/crio-b456e8e35c0f120681ba034e675eb1647a6833bc25a3e8850946e5f16a0f1370 WatchSource:0}: Error finding container b456e8e35c0f120681ba034e675eb1647a6833bc25a3e8850946e5f16a0f1370: Status 404 returned error can't find the container with id b456e8e35c0f120681ba034e675eb1647a6833bc25a3e8850946e5f16a0f1370
Oct 29 21:47:02.602694 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.602664    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4zxrn"]
Oct 29 21:47:02.605391 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:47:02.605355    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2a700a_5061_402a_ab9c_29a44204054c.slice/crio-c46ed14e7584251dfb42b440bc26db3bec43422182a35183f02aec5e8ed6b66d WatchSource:0}: Error finding container c46ed14e7584251dfb42b440bc26db3bec43422182a35183f02aec5e8ed6b66d: Status 404 returned error can't find the container with id c46ed14e7584251dfb42b440bc26db3bec43422182a35183f02aec5e8ed6b66d
Oct 29 21:47:02.625317 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.625284    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n8m99"]
Oct 29 21:47:02.627924 ip-10-0-136-38 kubenswrapper[2582]: W1029 21:47:02.627898    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fadb982_a597_4489_ae16_3dc4fb5a7810.slice/crio-cd556ab388fe7dcbbd7f3786761067d0275bc14e79b20048da9eb0c29c52f28f WatchSource:0}: Error finding container cd556ab388fe7dcbbd7f3786761067d0275bc14e79b20048da9eb0c29c52f28f: Status 404 returned error can't find the container with id cd556ab388fe7dcbbd7f3786761067d0275bc14e79b20048da9eb0c29c52f28f
Oct 29 21:47:02.662782 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.662745    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n8m99" event={"ID":"0fadb982-a597-4489-ae16-3dc4fb5a7810","Type":"ContainerStarted","Data":"cd556ab388fe7dcbbd7f3786761067d0275bc14e79b20048da9eb0c29c52f28f"}
Oct 29 21:47:02.664020 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.663989    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zxrn" event={"ID":"ef2a700a-5061-402a-ab9c-29a44204054c","Type":"ContainerStarted","Data":"c46ed14e7584251dfb42b440bc26db3bec43422182a35183f02aec5e8ed6b66d"}
Oct 29 21:47:02.665258 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:02.665225    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vth7d" event={"ID":"ae94d17c-b881-46e1-8063-e88c9dc53369","Type":"ContainerStarted","Data":"b456e8e35c0f120681ba034e675eb1647a6833bc25a3e8850946e5f16a0f1370"}
Oct 29 21:47:03.492828 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:03.492732    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:04.677375 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:04.677330    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zxrn" event={"ID":"ef2a700a-5061-402a-ab9c-29a44204054c","Type":"ContainerStarted","Data":"b84eb0dc43ea1320e4d01b4549219610957e7b54bff04549c5cf551024bcc03c"}
Oct 29 21:47:04.677854 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:04.677399    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4zxrn" event={"ID":"ef2a700a-5061-402a-ab9c-29a44204054c","Type":"ContainerStarted","Data":"1111971b90b971e62a1ebf5eba97e9c8d9293984b38f7b5f4c3cb1165f1e7337"}
Oct 29 21:47:04.722670 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:04.722594    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4zxrn" podStartSLOduration=67.66631798 podStartE2EDuration="1m8.722576107s" totalImagesPullingTime="1.056258127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:04.720920758 +0000 UTC m=+69.030595085" watchObservedRunningTime="2026-10-29 21:47:04.722576107 +0000 UTC m=+69.032250433"
Oct 29 21:47:06.689405 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:06.689359    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n8m99" event={"ID":"0fadb982-a597-4489-ae16-3dc4fb5a7810","Type":"ContainerStarted","Data":"25eb97730d986a07c418063e0c5a6e65e2e216370a435ff5853c940742c72c2f"}
Oct 29 21:47:06.689892 ip-10-0-136-38 kubenswrapper[2582]: I1029 21:47:06.689699    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:47:07.429986 ip-10-0-136-38 systemd[1]: Stopping Kubernetes Kubelet...
Oct 29 21:47:07.444217 ip-10-0-136-38 systemd[1]: kubelet.service: Deactivated successfully.
Oct 29 21:47:07.444399 ip-10-0-136-38 systemd[1]: Stopped Kubernetes Kubelet.
Oct 29 21:47:07.444458 ip-10-0-136-38 systemd[1]: kubelet.service: Consumed 5.073s CPU time.
Oct 29 21:47:07.454284 ip-10-0-136-38 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 21:47:07.533686 ip-10-0-136-38 kubenswrapper[7082]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:47:07.533686 ip-10-0-136-38 kubenswrapper[7082]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 21:47:07.533686 ip-10-0-136-38 kubenswrapper[7082]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:47:07.533686 ip-10-0-136-38 kubenswrapper[7082]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:47:07.535950 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535837    7082 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:47:07.535950 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535948    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:47:07.535950 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535951    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:47:07.535950 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535955    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:47:07.535950 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535957    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:47:07.535950 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535960    7082 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:47:07.535950 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535963    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535967    7082 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535970    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535972    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535975    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535977    7082 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535980    7082 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535982    7082 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535985    7082 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535988    7082 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535990    7082 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535993    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535995    7082 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.535998    7082 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536010    7082 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536013    7082 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536016    7082 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536020    7082 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536022    7082 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536025    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:47:07.536198 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536027    7082 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536030    7082 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536032    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536035    7082 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536037    7082 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536040    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536042    7082 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536045    7082 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536047    7082 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536050    7082 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536064    7082 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536069    7082 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536073    7082 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536076    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536079    7082 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536082    7082 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536085    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536089    7082 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536094    7082 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:47:07.536699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536097    7082 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536101    7082 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536104    7082 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536106    7082 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536109    7082 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536112    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536115    7082 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536117    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536120    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536123    7082 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536127    7082 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536130    7082 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536132    7082 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536135    7082 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536139    7082 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536142    7082 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536145    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536147    7082 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536150    7082 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:47:07.537157 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536153    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536155    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536158    7082 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536160    7082 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536163    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536166    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536168    7082 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536171    7082 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536174    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536177    7082 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536180    7082 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536182    7082 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536185    7082 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536187    7082 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536190    7082 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536193    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536195    7082 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536198    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536201    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:47:07.537619 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536206    7082 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536210    7082 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536213    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536216    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536219    7082 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536225    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536229    7082 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536231    7082 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536234    7082 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536237    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536240    7082 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536243    7082 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536245    7082 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536248    7082 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536251    7082 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536253    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536256    7082 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536258    7082 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536260    7082 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:47:07.538085 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536263    7082 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536265    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536268    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.536272    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537129    7082 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537138    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537141    7082 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537144    7082 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537147    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537149    7082 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537152    7082 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537155    7082 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537157    7082 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537160    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537163    7082 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537168    7082 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537171    7082 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537174    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537179    7082 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537183    7082 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:47:07.538531 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537185    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537188    7082 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537191    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537194    7082 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537196    7082 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537199    7082 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537201    7082 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537204    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537207    7082 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537209    7082 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537212    7082 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537215    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537217    7082 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537220    7082 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537223    7082 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537225    7082 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537229    7082 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537232    7082 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537235    7082 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537238    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:47:07.539044 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537240    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537243    7082 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537246    7082 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537249    7082 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537251    7082 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537254    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537256    7082 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537258    7082 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537261    7082 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537264    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537266    7082 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537269    7082 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537272    7082 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537275    7082 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537277    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537280    7082 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537282    7082 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537284    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537287    7082 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537290    7082 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:47:07.539530 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537292    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537296    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537298    7082 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537301    7082 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537303    7082 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537305    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537308    7082 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537311    7082 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537314    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537317    7082 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537319    7082 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537322    7082 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537324    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537328    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537330    7082 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537335    7082 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537338    7082 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537341    7082 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537343    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:47:07.540106 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537346    7082 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537348    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537351    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537354    7082 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537357    7082 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537359    7082 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537362    7082 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537364    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537367    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537369    7082 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537372    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537375    7082 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537377    7082 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537380    7082 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537382    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537384    7082 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537387    7082 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537389    7082 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537392    7082 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537394    7082 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:47:07.540611 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537401    7082 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537403    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537406    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537410    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537413    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537416    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537418    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537421    7082 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537423    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537426    7082 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.537428    7082 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538011    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538018    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538021    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538024    7082 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538026    7082 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538029    7082 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538033    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538035    7082 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:47:07.541148 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538038    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538041    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538043    7082 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538046    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538049    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538052    7082 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538054    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538057    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538059    7082 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538062    7082 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538064    7082 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538066    7082 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538069    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538073    7082 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538076    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538080    7082 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538089    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538093    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:47:07.541610 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538095    7082 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538098    7082 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538101    7082 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538103    7082 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538106    7082 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538109    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538112    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538114    7082 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538116    7082 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538119    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538122    7082 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538124    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538126    7082 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538129    7082 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538132    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538134    7082 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538137    7082 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538139    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538142    7082 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538144    7082 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:47:07.542069 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538147    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538149    7082 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538152    7082 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538154    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538157    7082 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538159    7082 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538162    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538166    7082 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538169    7082 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538171    7082 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538174    7082 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538182    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538186    7082 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538190    7082 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538193    7082 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538196    7082 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538200    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538203    7082 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538205    7082 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:47:07.542599 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538208    7082 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538210    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538213    7082 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538216    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538218    7082 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538221    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538223    7082 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538226    7082 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538228    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538231    7082 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538233    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538236    7082 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538238    7082 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538241    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538243    7082 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538246    7082 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538249    7082 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538252    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538254    7082 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538257    7082 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:47:07.543075 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538261    7082 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538264    7082 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538273    7082 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538275    7082 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538278    7082 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538287    7082 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538289    7082 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538292    7082 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538294    7082 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538298    7082 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538301    7082 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538304    7082 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538306    7082 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538309    7082 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538311    7082 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538314    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538316    7082 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538319    7082 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538321    7082 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538324    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.538326    7082 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:47:07.543558 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538442    7082 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538451    7082 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538459    7082 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538464    7082 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538468    7082 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538472    7082 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538477    7082 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538481    7082 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538484    7082 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538487    7082 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538491    7082 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538496    7082 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538499    7082 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538502    7082 flags.go:64] FLAG: --cgroup-root=""
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538505    7082 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538508    7082 flags.go:64] FLAG: --client-ca-file=""
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538511    7082 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538515    7082 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538520    7082 flags.go:64] FLAG: --cluster-domain=""
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538523    7082 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538527    7082 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538530    7082 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538534    7082 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538539    7082 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 21:47:07.544105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538542    7082 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538545    7082 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538549    7082 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538552    7082 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538555    7082 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538558    7082 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538561    7082 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538564    7082 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538569    7082 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538572    7082 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538575    7082 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538578    7082 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538581    7082 flags.go:64] FLAG: --enable-server="true"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538584    7082 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538589    7082 flags.go:64] FLAG: --event-burst="100"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538592    7082 flags.go:64] FLAG: --event-qps="50"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538595    7082 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538599    7082 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538602    7082 flags.go:64] FLAG: --eviction-hard=""
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538608    7082 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538611    7082 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538614    7082 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538618    7082 flags.go:64] FLAG: --eviction-soft=""
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538621    7082 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538639    7082 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 21:47:07.544700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538642    7082 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538646    7082 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538649    7082 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538654    7082 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538657    7082 flags.go:64] FLAG: --feature-gates=""
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538661    7082 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538665    7082 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538668    7082 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538672    7082 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538675    7082 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538678    7082 flags.go:64] FLAG: --help="false"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538681    7082 flags.go:64] FLAG: --hostname-override="ip-10-0-136-38.ec2.internal"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538685    7082 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538688    7082 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538691    7082 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538694    7082 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538698    7082 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538701    7082 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538704    7082 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538707    7082 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538709    7082 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538712    7082 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538716    7082 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538719    7082 flags.go:64] FLAG: --kube-reserved=""
Oct 29 21:47:07.545300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538722    7082 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538725    7082 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538729    7082 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538732    7082 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538735    7082 flags.go:64] FLAG: --lock-file=""
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538738    7082 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538741    7082 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538744    7082 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538751    7082 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538754    7082 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538766    7082 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538771    7082 flags.go:64] FLAG: --logging-format="text"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538774    7082 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538777    7082 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538780    7082 flags.go:64] FLAG: --manifest-url=""
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538783    7082 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538788    7082 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538791    7082 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538796    7082 flags.go:64] FLAG: --max-pods="110"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538799    7082 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538802    7082 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538805    7082 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538808    7082 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538811    7082 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538814    7082 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 21:47:07.545942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538817    7082 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538825    7082 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538828    7082 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538831    7082 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538834    7082 flags.go:64] FLAG: --pod-cidr=""
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538837    7082 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538840    7082 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538843    7082 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538846    7082 flags.go:64] FLAG: --port="10250"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538851    7082 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538854    7082 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b2927172f04d03f2"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538858    7082 flags.go:64] FLAG: --qos-reserved=""
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538861    7082 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538864    7082 flags.go:64] FLAG: --register-node="true"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538867    7082 flags.go:64] FLAG: --register-with-taints=""
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538871    7082 flags.go:64] FLAG: --registry-burst="10"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538874    7082 flags.go:64] FLAG: --registry-qps="5"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538877    7082 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538888    7082 flags.go:64] FLAG: --reserved-memory=""
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538893    7082 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538897    7082 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538901    7082 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538903    7082 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538906    7082 flags.go:64] FLAG: --runonce="false"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538910    7082 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538913    7082 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538916    7082 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 21:47:07.546532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538919    7082 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538922    7082 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538925    7082 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538928    7082 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538931    7082 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538934    7082 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538937    7082 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538940    7082 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538943    7082 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538946    7082 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538949    7082 flags.go:64] FLAG: --system-cgroups=""
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538952    7082 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538958    7082 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538961    7082 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538965    7082 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.538998    7082 flags.go:64] FLAG: --tls-min-version=""
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.539001    7082 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.539004    7082 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.539007    7082 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.539010    7082 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.539013    7082 flags.go:64] FLAG: --v="2"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.539024    7082 flags.go:64] FLAG: --version="false"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.539029    7082 flags.go:64] FLAG: --vmodule=""
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.539035    7082 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.539040    7082 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 21:47:07.547573 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539193    7082 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539197    7082 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539200    7082 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539203    7082 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539206    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539208    7082 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539211    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539213    7082 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539216    7082 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539219    7082 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539221    7082 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539224    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539227    7082 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539229    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539232    7082 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539234    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539237    7082 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539240    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539242    7082 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539245    7082 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:47:07.548392 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539248    7082 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539253    7082 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539255    7082 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539258    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539261    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539263    7082 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539266    7082 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539269    7082 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539271    7082 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539274    7082 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539278    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539280    7082 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539284    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539286    7082 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539289    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539291    7082 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539294    7082 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539297    7082 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539299    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:47:07.548978 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539302    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539304    7082 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539307    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539309    7082 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539312    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539314    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539317    7082 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539319    7082 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539322    7082 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539324    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539326    7082 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539329    7082 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539332    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539334    7082 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539338    7082 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539341    7082 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539344    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539346    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539349    7082 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:47:07.549435 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539352    7082 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539356    7082 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539360    7082 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539363    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539367    7082 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539370    7082 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539373    7082 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539379    7082 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539382    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539385    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539388    7082 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539390    7082 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539393    7082 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539395    7082 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539398    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539402    7082 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539405    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539408    7082 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539411    7082 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:47:07.549933 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539413    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539416    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539418    7082 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539420    7082 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539423    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539426    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539428    7082 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539431    7082 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539435    7082 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539438    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539440    7082 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539443    7082 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539446    7082 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539448    7082 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539451    7082 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539454    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539456    7082 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539460    7082 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539463    7082 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:47:07.550375 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539466    7082 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539468    7082 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539475    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539478    7082 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539480    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539482    7082 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539485    7082 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539487    7082 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539490    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539493    7082 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.539521    7082 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539560    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539565    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539568    7082 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539571    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539574    7082 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:47:07.550855 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539576    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539578    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539581    7082 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539583    7082 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539586    7082 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539588    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539591    7082 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539594    7082 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539597    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539599    7082 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539602    7082 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539604    7082 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539607    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539609    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539612    7082 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539615    7082 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539619    7082 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539637    7082 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539642    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:47:07.551338 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539645    7082 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539649    7082 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539652    7082 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539654    7082 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539657    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539659    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539662    7082 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539665    7082 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539667    7082 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539670    7082 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539672    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539676    7082 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539679    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539682    7082 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539687    7082 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539692    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539695    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539700    7082 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539704    7082 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539707    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:47:07.551874 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539711    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539715    7082 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539719    7082 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539723    7082 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539727    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539730    7082 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539733    7082 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539736    7082 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539739    7082 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539741    7082 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539744    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539746    7082 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539749    7082 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539752    7082 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539755    7082 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539758    7082 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539763    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539765    7082 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539768    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:47:07.552416 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539770    7082 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539773    7082 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539775    7082 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539778    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539781    7082 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539784    7082 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539789    7082 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539794    7082 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539799    7082 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539803    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539807    7082 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539810    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539813    7082 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539815    7082 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539818    7082 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539821    7082 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539824    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539827    7082 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539829    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539832    7082 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:47:07.552989 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539834    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539837    7082 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539839    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539842    7082 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539844    7082 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539846    7082 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539849    7082 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539852    7082 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539855    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539858    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539860    7082 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539862    7082 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539865    7082 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539868    7082 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539873    7082 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539877    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539882    7082 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539886    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539891    7082 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539894    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:47:07.554145 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539896    7082 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539900    7082 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.539903    7082 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.544756    7082 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.544779    7082 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.544800    7082 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.544806    7082 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544892    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544899    7082 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544903    7082 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544905    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544908    7082 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544911    7082 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544914    7082 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544916    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544919    7082 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544923    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544926    7082 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544928    7082 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544937    7082 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:47:07.554974 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544940    7082 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544943    7082 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544946    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544948    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544951    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544954    7082 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544957    7082 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544959    7082 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544963    7082 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544968    7082 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544972    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544975    7082 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544978    7082 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544981    7082 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544983    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544986    7082 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544989    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544991    7082 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544994    7082 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:47:07.555561 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.544997    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545000    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545002    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545005    7082 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545007    7082 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545010    7082 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545013    7082 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545016    7082 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545018    7082 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545021    7082 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545024    7082 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545027    7082 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545029    7082 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545040    7082 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545043    7082 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545046    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545049    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545051    7082 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545054    7082 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545057    7082 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:47:07.556133 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545060    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545063    7082 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545065    7082 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545068    7082 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545071    7082 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545073    7082 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545076    7082 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545078    7082 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545080    7082 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545083    7082 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545086    7082 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545088    7082 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545090    7082 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545093    7082 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545095    7082 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545098    7082 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545100    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545103    7082 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545106    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545108    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:47:07.556710 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545111    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545114    7082 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545117    7082 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545119    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545122    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545124    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545133    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545137    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545139    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545142    7082 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545144    7082 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545147    7082 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545150    7082 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545152    7082 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545155    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545157    7082 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545160    7082 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545164    7082 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545167    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:47:07.557180 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545171    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545174    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545176    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545179    7082 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545182    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545184    7082 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545187    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545189    7082 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545191    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545194    7082 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545197    7082 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545199    7082 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545209    7082 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545212    7082 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545215    7082 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.545248    7082 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:47:07.557637 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545457    7082 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545463    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545466    7082 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545469    7082 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545478    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545481    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545484    7082 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545486    7082 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545489    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545492    7082 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545494    7082 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545497    7082 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545499    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545502    7082 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545504    7082 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545507    7082 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545509    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545512    7082 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545515    7082 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:47:07.558063 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545518    7082 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545520    7082 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545523    7082 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545525    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545528    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545531    7082 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545533    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545535    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545538    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545541    7082 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545544    7082 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545546    7082 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545549    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545552    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545555    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545558    7082 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545562    7082 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545569    7082 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545577    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545580    7082 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:47:07.558673 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545583    7082 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545585    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545588    7082 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545592    7082 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545596    7082 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545599    7082 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545601    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545604    7082 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545606    7082 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545609    7082 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545611    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545614    7082 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545616    7082 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545619    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545621    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545643    7082 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545646    7082 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545649    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545652    7082 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545655    7082 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:47:07.559178 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545658    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545661    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545664    7082 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545666    7082 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545669    7082 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545671    7082 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545674    7082 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545677    7082 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545679    7082 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545682    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545684    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545694    7082 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545697    7082 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545700    7082 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545702    7082 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545704    7082 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545707    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545709    7082 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545712    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:47:07.559700 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545714    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545717    7082 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545719    7082 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545722    7082 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545724    7082 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545727    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545729    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545732    7082 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545734    7082 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545740    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545742    7082 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545745    7082 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545748    7082 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545751    7082 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545753    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545756    7082 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545758    7082 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545761    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545763    7082 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545766    7082 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:47:07.560209 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545769    7082 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545771    7082 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545774    7082 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545776    7082 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545779    7082 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545787    7082 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545790    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545792    7082 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.545817    7082 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545865    7082 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545870    7082 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545873    7082 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545876    7082 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545879    7082 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545881    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545884    7082 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:47:07.560808 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545886    7082 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545889    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545891    7082 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545896    7082 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545899    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545901    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545906    7082 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545908    7082 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545911    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545914    7082 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545916    7082 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545919    7082 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545921    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545924    7082 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545926    7082 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545929    7082 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545931    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545934    7082 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545936    7082 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:47:07.561194 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545939    7082 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545941    7082 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545943    7082 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545952    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545955    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545958    7082 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545960    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545963    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545965    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545968    7082 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545970    7082 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545973    7082 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545975    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545978    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545980    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545984    7082 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545988    7082 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545991    7082 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545994    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.545998    7082 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:47:07.561699 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546000    7082 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546003    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546005    7082 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546009    7082 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546012    7082 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546016    7082 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546019    7082 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546022    7082 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546025    7082 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546028    7082 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546031    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546033    7082 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546036    7082 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546038    7082 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546041    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546043    7082 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546051    7082 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546054    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546057    7082 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546059    7082 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:47:07.562199 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546061    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546064    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546066    7082 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546069    7082 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546071    7082 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546074    7082 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546076    7082 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546079    7082 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546081    7082 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546084    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546086    7082 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546089    7082 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546091    7082 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546094    7082 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546096    7082 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546099    7082 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546101    7082 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546103    7082 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546106    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:47:07.562728 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546108    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546111    7082 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546113    7082 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546115    7082 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546118    7082 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546120    7082 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546123    7082 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546125    7082 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546128    7082 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546130    7082 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546138    7082 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546141    7082 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546143    7082 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546146    7082 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546149    7082 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546151    7082 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546153    7082 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546155    7082 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546158    7082 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546160    7082 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:47:07.563261 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:07.546164    7082 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:47:07.563771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.546223    7082 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 21:47:07.563771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.548585    7082 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 29 21:47:07.563771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.548689    7082 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 29 21:47:07.563771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.548935    7082 server.go:1014] "Starting client certificate rotation"
Oct 29 21:47:07.563771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.549129    7082 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 21:47:07.563771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.549187    7082 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 21:40:56 +0000 UTC" deadline="2028-06-02 02:38:26.501554985 +0000 UTC"
Oct 29 21:47:07.563771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.549261    7082 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13948h51m18.952297518s"
Oct 29 21:47:07.563771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.549602    7082 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:47:07.563771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.551061    7082 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:47:07.563771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.552411    7082 log.go:25] "Validated CRI v1 runtime API"
Oct 29 21:47:07.563771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.555117    7082 log.go:25] "Validated CRI v1 image API"
Oct 29 21:47:07.563771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.555957    7082 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 21:47:07.563771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.558787    7082 fs.go:135] Filesystem UUIDs: map[29e4a222-747d-4d84-b4fd-49720bd6275f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d0dd7a92-3704-452c-b857-d6829f3b88b5:/dev/nvme0n1p3]
Oct 29 21:47:07.564148 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.558819    7082 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b5c45168936ec68706fa22cf5010febbd2abb5b13240dcbafa697326f78db6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b5c45168936ec68706fa22cf5010febbd2abb5b13240dcbafa697326f78db6f/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1dd1bcbd88133555363bf351bf334c8198266dea4c32c7054d43127336b3f5cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1dd1bcbd88133555363bf351bf334c8198266dea4c32c7054d43127336b3f5cc/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b7ac84a5de67fab87859f8db39452d0ad41c49da9b34418888d7ff01e658423/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b7ac84a5de67fab87859f8db39452d0ad41c49da9b34418888d7ff01e658423/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2deaa56200d86b0a245d5d869577bf52e61f8ae0b923ff824368b1099a340f41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2deaa56200d86b0a245d5d869577bf52e61f8ae0b923ff824368b1099a340f41/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32380f5c9fc4b9aa3b0dfab906c26f7d8179c9ff63f0b35e0533089efb12c27b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32380f5c9fc4b9aa3b0dfab906c26f7d8179c9ff63f0b35e0533089efb12c27b/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3239833dfc0a5983fdfc923651715e570c2cc3757bc1eaf4131ee67a1750d4d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3239833dfc0a5983fdfc923651715e570c2cc3757bc1eaf4131ee67a1750d4d3/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3259f5e5de34085e454c4fec63e4c26517488e625b618e564cdc7fc4f3ffe1d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3259f5e5de34085e454c4fec63e4c26517488e625b618e564cdc7fc4f3ffe1d9/userdata/shm major:0 minor:386 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d812d8da771abbedc07f0f5ed688a977dbdce8dae239dad6e12b15651d5b83b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d812d8da771abbedc07f0f5ed688a977dbdce8dae239dad6e12b15651d5b83b/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41b90bddf204131ae88a459287592030baa4a2335b039cc08b45c4d34dc96c8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41b90bddf204131ae88a459287592030baa4a2335b039cc08b45c4d34dc96c8f/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b7b55989043bc79ff2d50ed75a4b5de802ae3abe4a56324e8389bd9efd54e44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b7b55989043bc79ff2d50ed75a4b5de802ae3abe4a56324e8389bd9efd54e44/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b96159b5793d1be00bf8b618ba9cdf13625516b0c1b39e7ccea9a4794d29467/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b96159b5793d1be00bf8b618ba9cdf13625516b0c1b39e7ccea9a4794d29467/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60374fa76ee4647cf89872d467fec40c6a20477a19eccc4a213e21518f86c57a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60374fa76ee4647cf89872d467fec40c6a20477a19eccc4a213e21518f86c57a/userdata/shm major:0 minor:480 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64227e95b780817a73780d1bd50fe0918a17f696017280aff7ff4d040aed8d6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64227e95b780817a73780d1bd50fe0918a17f696017280aff7ff4d040aed8d6c/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66df3707a72a218383de890083a85d74eae68aee8ee9d173562b1d62817fae0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66df3707a72a218383de890083a85d74eae68aee8ee9d173562b1d62817fae0a/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c733459254465418be5838dcabb8ad525786cf2728369479663d2335fd3dee3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c733459254465418be5838dcabb8ad525786cf2728369479663d2335fd3dee3/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70b655ab16946107983c75a25d1a564c589aa3d4fc7751290f65f274ca5a4122/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70b655ab16946107983c75a25d1a564c589aa3d4fc7751290f65f274ca5a4122/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/930a013f9bff036cccddc08345d999cc49d1c9bde3b77806a37c98ec3fc82c98/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/930a013f9bff036cccddc08345d999cc49d1c9bde3b77806a37c98ec3fc82c98/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95e96df2fcf5249c4d02b0c62687b1782bb6e330bf154c7116adfc74f9765753/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95e96df2fcf5249c4d02b0c62687b1782bb6e330bf154c7116adfc74f9765753/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9999ce6f5d6e9d98d8a8f249aa37b8b9da08fa960a1f6bcf5c7b41c34d33db18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9999ce6f5d6e9d98d8a8f249aa37b8b9da08fa960a1f6bcf5c7b41c34d33db18/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a727e03aa88193020fda425d0065af63e966b5a3dfc59c8264358d9d7dc7c986/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a727e03aa88193020fda425d0065af63e966b5a3dfc59c8264358d9d7dc7c986/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b456e8e35c0f120681ba034e675eb1647a6833bc25a3e8850946e5f16a0f1370/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b456e8e35c0f120681ba034e675eb1647a6833bc25a3e8850946e5f16a0f1370/userdata/shm major:0 minor:493 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bed90a2833cf42ce4a410e6c930d6ef7667ac810586d5f076818e1b5ecaad60f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bed90a2833cf42ce4a410e6c930d6ef7667ac810586d5f076818e1b5ecaad60f/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c46ed14e7584251dfb42b440bc26db3bec43422182a35183f02aec5e8ed6b66d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c46ed14e7584251dfb42b440bc26db3bec43422182a35183f02aec5e8ed6b66d/userdata/shm major:0 minor:495 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd556ab388fe7dcbbd7f3786761067d0275bc14e79b20048da9eb0c29c52f28f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd556ab388fe7dcbbd7f3786761067d0275bc14e79b20048da9eb0c29c52f28f/userdata/shm major:0 minor:497 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d78fcfe0ae169d0d24484aec02afab6b7850f1079511868ac83f9d06b3b590b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d78fcfe0ae169d0d24484aec02afab6b7850f1079511868ac83f9d06b3b590b8/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbf589c6e0dabaec0d066764594086a44f6bc30d6f4d74670ad6dca76ed2ad62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbf589c6e0dabaec0d066764594086a44f6bc30d6f4d74670ad6dca76ed2ad62/userdata/shm major:0 minor:467 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e01d005eae6c14c03a9634425443b9564b8d5a3e28ded5ea1fc6347a08fec8ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e01d005eae6c14c03a9634425443b9564b8d5a3e28ded5ea1fc6347a08fec8ba/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/066041b1-7eba-4371-9470-101a7bf8fcfc/volumes/kubernetes.io~projected/kube-api-access-b7l79:{mountpoint:/var/lib/kubelet/pods/066041b1-7eba-4371-9470-101a7bf8fcfc/volumes/kubernetes.io~projected/kube-api-access-b7l79 major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/066041b1-7eba-4371-9470-101a7bf8fcfc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/066041b1-7eba-4371-9470-101a7bf8fcfc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/066041b1-7eba-4371-9470-101a7bf8fcfc/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/066041b1-7eba-4371-9470-101a7bf8fcfc/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fadb982-a597-4489-ae16-3dc4fb5a7810/volumes/kubernetes.io~projected/kube-api-access-ht42j:{mountpoint:/var/lib/kubelet/pods/0fadb982-a597-4489-ae16-3dc4fb5a7810/volumes/kubernetes.io~projected/kube-api-access-ht42j major:0 minor:492 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f41e102-fab5-4206-9502-fbf09515069a/volumes/kubernetes.io~projected/kube-api-access-8vrkn:{mountpoint:/var/lib/kubelet/pods/1f41e102-fab5-4206-9502-fbf09515069a/volumes/kubernetes.io~projected/kube-api-access-8vrkn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f64c76d-6e73-4cc0-9043-8f6235e0e281/volumes/kubernetes.io~projected/kube-api-access-sz6vz:{mountpoint:/var/lib/kubelet/pods/2f64c76d-6e73-4cc0-9043-8f6235e0e281/volumes/kubernetes.io~projected/kube-api-access-sz6vz major:0 minor:479 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f64c76d-6e73-4cc0-9043-8f6235e0e281/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/2f64c76d-6e73-4cc0-9043-8f6235e0e281/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:473 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f64c76d-6e73-4cc0-9043-8f6235e0e281/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/2f64c76d-6e73-4cc0-9043-8f6235e0e281/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:478 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3017c536-859c-4b7b-b59e-f1d5ee77e117/volumes/kubernetes.io~projected/kube-api-access-zvtbk:{mountpoint:/var/lib/kubelet/pods/3017c536-859c-4b7b-b59e-f1d5ee77e117/volumes/kubernetes.io~projected/kube-api-access-zvtbk major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volume-subpaths/web-config/alertmanager/9 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volume-subpaths/web-config/config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~empty-dir/config-out major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~projected/kube-api-access-klh9r:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~projected/kube-api-access-klh9r major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~projected/tls-assets major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/config-volume major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/web-config major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/324d72a4-a641-4b09-99d6-d682b5e2c958/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/324d72a4-a641-4b09-99d6-d682b5e2c958/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3accc688-8a04-4462-9593-228a18548478/volumes/kubernetes.io~projected/kube-api-access-gnclr:{mountpoint:/var/lib/kubelet/pods/3accc688-8a04-4462-9593-228a18548478/volumes/kubernetes.io~projected/kube-api-access-gnclr major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3accc688-8a04-4462-9593-228a18548478/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/3accc688-8a04-4462-9593-228a18548478/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b64908a-fffb-4ec2-8697-51ea5db37f3b/volumes/kubernetes.io~projected/kube-api-access-bkhr5:{mountpoint:/var/lib/kubelet/pods/3b64908a-fffb-4ec2-8697-51ea5db37f3b/volumes/kubernetes.io~projected/kube-api-access-bkhr5 major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b64908a-fffb-4ec2-8697-51ea5db37f3b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3b64908a-fffb-4ec2-8697-51ea5db37f3b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b64908a-fffb-4ec2-8697-51ea5db37f3b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/3b64908a-fffb-4ec2-8697-51ea5db37f3b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c2dd590-c639-42fe-8a02-4945b6403b3b/volumes/kubernetes.io~projected/kube-api-access-t5fvk:{mountpoint:/var/lib/kubelet/pods/4c2dd590-c639-42fe-8a02-4945b6403b3b/volumes/kubernetes.io~projected/kube-api-access-t5fvk major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c2dd590-c639-42fe-8a02-4945b6403b3b/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/4c2dd590-c639-42fe-8a02-4945b6403b3b/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5183b0be-c47a-4e35-a8cb-77991e6f7208/volumes/kubernetes.io~projected/kube-api-access-zpbcb:{mountpoint:/var/lib/kubelet/pods/5183b0be-c47a-4e35-a8cb-77991e6f7208/volumes/kubernetes.io~projected/kube-api-access-zpbcb major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79bf4395-e3d6-4071-aedd-548c185fa39d/volumes/kubernetes.io~projected/kube-api-access-kmrsr:{mountpoint:/var/lib/kubelet/pods/79bf4395-e3d6-4071-aedd-548c185fa39d/volumes/kubernetes.io~projected/kube-api-access-kmrsr major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79bf4395-e3d6-4071-aedd-548c185fa39d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/79bf4395-e3d6-4071-aedd-548c185fa39d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/801f804c-187a-4232-af07-412a115a2fef/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/801f804c-187a-4232-af07-412a115a2fef/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/801f804c-187a-4232-af07-412a115a2fef/volumes/kubernetes.io~projected/kube-api-access-7rfd8:{mountpoint:/var/lib/kubelet/pods/801f804c-187a-4232-af07-412a115a2fef/volumes/kubernetes.io~projected/kube-api-access-7rfd8 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/801f804c-187a-4232-af07-412a115a2fef/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/801f804c-187a-4232-af07-412a115a2fef/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c893b43-cd05-45a0-9f7b-a2b5d72809c7/volumes/kubernetes.io~projected/kube-api-access-ldwrg:{mountpoint:/var/lib/kubelet/pods/8c893b43-cd05-45a0-9f7b-a2b5d72809c7/volumes/kubernetes.io~projected/kube-api-access-ldwrg major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c893b43-cd05-45a0-9f7b-a2b5d72809c7/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8c893b43-cd05-45a0-9f7b-a2b5d72809c7/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c893b43-cd05-45a0-9f7b-a2b5d72809c7/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/8c893b43-cd05-45a0-9f7b-a2b5d72809c7/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a09b130d-4249-4539-add0-e0aba69b752a/volumes/kubernetes.io~projected/kube-api-access-rgf24:{mountpoint:/var/lib/kubelet/pods/a09b130d-4249-4539-add0-e0aba69b752a/volumes/kubernetes.io~projected/kube-api-access-rgf24 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f/volumes/kubernetes.io~projected/kube-api-access-drx2d:{mountpoint:/var/lib/kubelet/pods/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f/volumes/kubernetes.io~projected/kube-api-access-drx2d major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae94d17c-b881-46e1-8063-e88c9dc53369/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ae94d17c-b881-46e1-8063-e88c9dc53369/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:491 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~projected/kube-api-access-vv8bm:{mountpoint:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~projected/kube-api-access-vv8bm major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baaab207-5534-4ef9-ac3a-9466ea5b900e/volumes/kubernetes.io~projected/kube-api-access-hkc5w:{mountpoint:/var/lib/kubelet/pods/baaab207-5534-4ef9-ac3a-9466ea5b900e/volumes/kubernetes.io~projected/kube-api-access-hkc5w major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baaab207-5534-4ef9-ac3a-9466ea5b900e/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/baaab207-5534-4ef9-ac3a-9466ea5b900e/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baaab207-5534-4ef9-ac3a-9466ea5b900e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/baaab207-5534-4ef9-ac3a-9466ea5b900e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baaab207-5534-4ef9-ac3a-9466ea5b900e/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/baaab207-5534-4ef9-ac3a-9466ea5b900e/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe53da9-f387-477c-8591-f5c37f42eaf5/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/bbe53da9-f387-477c-8591-f5c37f42eaf5/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe53da9-f387-477c-8591-f5c37f42eaf5/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/bbe53da9-f387-477c-8591-f5c37f42eaf5/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe53da9-f387-477c-8591-f5c37f42eaf5/volumes/kubernetes.io~projected/kube-api-access-v6m9n:{mountpoint:/var/lib/kubelet/pods/bbe53da9-f387-477c-8591-f5c37f42eaf5/volumes/kubernetes.io~projected/kube-api-access-v6m9n major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c20e4d88-bb47-4294-8242-2e71ca6d3f6c/volumes/kubernetes.io~projected/kube-api-access-vcnrm:{mountpoint:/var/lib/kubelet/pods/c20e4d88-bb47-4294-8242-2e71ca6d3f6c/volumes/kubernetes.io~projected/kube-api-access-vcnrm major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c20e4d88-bb47-4294-8242-2e71ca6d3f6c/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/c20e4d88-bb47-4294-8242-2e71ca6d3f6c/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:465 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c20e4d88-bb47-4294-8242-2e71ca6d3f6c/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/c20e4d88-bb47-4294-8242-2e71ca6d3f6c/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:461 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2e1c568-867f-472f-9c39-47588d5f3fca/volumes/kubernetes.io~projected/kube-api-access-vshbg:{mountpoint:/var/lib/kubelet/pods/c2e1c568-867f-472f-9c39-47588d5f3fca/volumes/kubernetes.io~projected/kube-api-access-vshbg major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd3c4499-8959-4616-842b-c556e7d23137/volumes/kubernetes.io~projected/kube-api-access-b6b42:{mountpoint:/var/lib/kubelet/pods/cd3c4499-8959-4616-842b-c556e7d23137/volumes/kubernetes.io~projected/kube-api-access-b6b42 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd3c4499-8959-4616-842b-c556e7d23137/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/cd3c4499-8959-4616-842b-c556e7d23137/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~projected/kube-api-access-t9d2j:{mountpoint:/var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~projected/kube-api-access-t9d2j major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef2a700a-5061-402a-ab9c-29a44204054c/volumes/kubernetes.io~projected/kube-api-access-tbtl8:{mountpoint:/var/lib/kubelet/pods/ef2a700a-5061-402a-ab9c-29a44204054c/volumes/kubernetes.io~projected/kube-api-access-tbtl8 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef2a700a-5061-402a-ab9c-29a44204054c/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/ef2a700a-5061-402a-ab9c-29a44204054c/volumes/kubernetes.io~secret/metrics-certs major:0 minor:486 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f683dbce-ebac-4d2c-ba04-aba3ac20792b/volumes/kubernetes.io~projected/kube-api-access-kck4b:{mountpoint:/var/lib/kubelet/pods/f683dbce-ebac-4d2c-ba04-aba3ac20792b/volumes/kubernetes.io~projected/kube-api-access-kck4b major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c96f87e8dfdb4eb8ca49fa22db98ef6b3ed1202bdfac1a3f37dc71c8477210e3/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6d42a28cf861b642ba4bc06f3c1a0130d28d681e18e600fb2af5a044adf48411/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3eef5983e5b58a66891288d39a7b06018b7737ce0268aae78109907d61a7a924/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/5a1d520a71bd5be6625c336301a4a5c22f88239c319047e90cc898b85c1ada87/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/41de065cfc10e06e76f753ea079dd91fe532cfcdb36e75826dd986006a925b7a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ea7cb5e4d409e7faa0f665ba4c5171a0f9a5c815aef8cb594a9425c4cbde4fdf/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/499904119d88e1ef3520f7c4ece2fb323b27923649d373d8a665d20dfa7b2a60/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/823156e2ea0b5f9042f9892c60ce5497a463ee31d98d266864e976ec42159514/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/7a718033f8d571596f62556a2c3b40252cc913ab4c85536e79dff3703775d647/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/eafbde367c60e934e00106f0d2a68149c6c2059c50a34e47b560716702e98915/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/862f2b2cba49f9fb142d99c2a13d3c3803204d6bb3c1df6eb5e6754df7205d99/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/c5cf8b9f7a36208e47beae605a78e8085e48fe792ddad1433c4f08adef2ba015/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/4b3d0e9ee1788f1aba6e4ba96d0125d9ef0e3b1b8240df9085bafa17932fc044/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7989e60c582f572f461e8685a0761dcc0041128cf10c2c44de4d175da79e26b8/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d680aecd996fcaed785a8a61b443a788d0ee1b4e51893ea9bd226407d50d036a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/20dc850cfa015e1e52318482efa9c27096e0bc36d965af9b2708698e9b4e4291/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d9696e3a1398cdca3cdb4cde03f35eec881688dbf4f8dcbed5fb465d450fd4c4/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0ec0e31cabfb28a7b957e19100eb32434c363d82921ae1183814cc87bc53d2c9/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2347a383d05579172c7f5915bd7360ca8b282485e3ca52d785d4190198687b91/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/df2e224c0ed7044efddcc05a2c6fc0af61d3e3025fd05e25a364685949febddf/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8fc2c6e6e2e78ead33ca917b72958601d1777888755f74a69e51da2e44857dd3/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/da9640e2b5140fb09301f87ca38db98c43bf5063f5be0f64b6de62d7aaf95220/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/395ec44032f954937668f13f41d78f0ea2453ffd9354629a2541dcd453ed36d5/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/2cc7a42713ed864368c9745b32c5cfead5f02b67542359ed690c679738d6b6b7/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/af6b4b07d27749e3d5723cf62d0bad5c410317030f5c96b92645692441636fe7/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/f8f56aa9dd184171f3e012e4d3089d4bc797154600545c073b0e089bd3762665/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/029f5a8baf7ef361efb18832a24f5b71d392e83ab01975da67edcd7d664d27ac/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/28d55dd6f6fb59ae12a6916ba4004ec2a08667869c165f2d9bcc59ff70cf9188/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/13ee83133a5fa86a829c4f9c48bf3fe390c624b4a12d0c10ab5a4df98dc3aac5/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/7c77fb79af94a6539e27347b8d63f6b557a4fe985f9051608432f94184633ab9/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/1c6283c17fd5d689fc6496ff15ad7e5292807cf9138bd89ec7aaa995739871eb/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/ff86e2d515917fb7b870d21404561941e80b3795ae56b9594d88b7396bbe5795/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/a6cb7c469d2bdde759c7a0cbe39ecac6a87002b2cf0e14ed56be05abb4bb9eb2/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/1684c95a730744aa16084c3edd8950786a861a569f1cf805f2f26bec86304baf/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/76c44e526ac832272d3f739b22aac71b91f82da99551d123d7f740575c232923/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/cc8922cbd55205d8c216a0c8350c49e5fff6b9da385ab148bd34cf6b1ea1ddbb/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/494c5af9b22d1481e40ea0092a58718b8636a935decce497fcf3f9efa82c1b0e/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/cce73e5b4f934be055d08520f7c7ff121159d6624738bd9846acfdbdabd32481/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/7daa39a208bc852418a43e0141735a918641cfc1d11735f97583efc97ca1bf10/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/adf7218733f1e5fa6dbb8c0e91b96f6f560926115a8b75da993b02550a4446d2/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/9897fa798adf0b887467e752d3a243d5232dfac2950acc532233b91831ad2955/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/e6160a6ec3d8afb54c523726eee9707bec2bc7cebc581bf2405b6c71b9fb5b14/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/175be1103f2ee140ddad7259bfe0325e068e062012b9da9f98b28455fc940815/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/8e193bf1e3033d491daf91dc88813d6bbda392775536624062ee44757d03d7d0/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/4843b7ec468ca93d35ef1ff7d2e0ae84ebb3602fdfbbf5fb97b4fde91295fa70/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/76096da963e64da254f678604b68229295fda7f1b50e96f0db56fa24e36ea18e/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/07c7ec972c6acc53d9c9b8b3d38d6c866eb67c5b771ca309c0bd41f4a4d282aa/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/4bdf6be06879d4fef0d5f815e38284f956c29f0af8db3b7c646f790d8b174abe/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/0c804bf029ccdb0c53d2786145f01158973ae81ed4910edbca1cc8eeef8b6ea9/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/c2ca883e49c5b818a115e192a76b6135e4fa27020665d553c4f73a81672f4fb1/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/c2f8bb569f87bd48343e16d672873f89f64253d208619fa865193c66b6d39efa/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/8f7d81e4dc3411674d1c71d1eda7ce9acd95af7017e48611d3bef87eabab574f/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/0f40ab6f09d416cbe3854a3898fcfceb99184a1a7c3ef458d91f2187cb315717/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/f62a07ce86e5ab6ebc55718b8215808b8ad2bf870da05d7d05c8ee39fc1033e8/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/15ee65adf383cd4c340e478f1c035a8ad6223cbc69df64c4046775b0faa394fe/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/019fc1c442f44ed479243df734f4c23e8be5e405c6dde25360337462d48b6dab/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/6d46c96ce2443598e6abb670613d5f7f03dfde7ab3bc77ab7d0a4c0e15a5e098/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/97b7c08727cd5f0256115e14ad16fe2242a730663c0bdab8b0e0506834104007/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/40bbbc610457aa208d9393b3805464c1bb256d1573f24bd99cb0e543fbf9edb4/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/8cc386914d322f0e08c1db7fae4c6e4a635ca993f409fde10bc5036a9001b870/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-433:{mountpoint:/var/lib/containers/storage/overlay/5e039f25d4b69da4d528d012c81c4ee080eee41055c717dd434baee26e7f6f2f/merged major:0 minor:433 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/d19e4e170bf465c6d1ecd38d93de40132f50594784e0376c1915e38a17d5e23e/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/52816513ccf7d01733ec1d18a4bbf1e36242fa952a32716abf105905b41976c1/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/99601e6ada822030b941736fa3d2e07cbd8592c04fc69abfd18ee6685a06de02/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/b6f8f3c2556d89c1ff34ed1e7a52b81f84ceef8fc90ed8db7f3820f7c8eef970/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/cc04f5c08134fa4477b36694b9ab28e9cbf0c38c21189a4115c1a9f356318e06/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-471:{mountpoint:/var/lib/containers/storage/overlay/a64473e42b5161d89bedc05ca9ad4e71ae3963d18a451c8deb41411325ff3812/merged major:0 minor:471 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/563ea18b93fc501b2228f9bab428665c1bb3423c1c5772b2d9e4ffed3273b87b/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/81cc0c62e81c4dc7b3b92b62b082eb920b5a285e28e7e446d4ab4c82c4a13fb8/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-499:{mountpoint:/var/lib/containers/storage/overlay/caf21382ce4bfe417c814eada36eab8e1a8181a8ab3cf727adde9bad89b9922e/merged major:0 minor:499 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/e00ed6f7e3246eb1ba5f2e75ff47321f91fed1fc93d6a4b6d4b36410abc54966/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/51d6a1167f8955b4036d5ff24be2f725ca11572ee163d856f637dfaf065ed6ab/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-505:{mountpoint:/var/lib/containers/storage/overlay/f1de0dd5845a0249eb57540359fb3073525856975558cdc40bb12b3e127ec204/merged major:0 minor:505 fsType:overlay blockSize:0} overlay_0-507:{mountpoint:/var/lib/containers/storage/overlay/befd3a8894a545d22b96a4ebd16d677697aa4d2490b4c8ea1adc610369313fe6/merged major:0 minor:507 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/e3d40403b97965045745e62613acc8604f3923c8dec870c80188cbaf319811d7/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-513:{mountpoint:/var/lib/containers/storage/overlay/67c1865571a924c15e9c00e50ca1d3e8f9ca02558858305c505505cdbb321df1/merged major:0 minor:513 fsType:overlay blockSize:0} overlay_0-518:{mountpoint:/var/lib/containers/storage/overlay/f63c250f662464c3e3e811469e63052a9804e12393ac0bbe1f23e03ac6d42252/merged major:0 minor:518 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/7f5afc93218b9391b2df4d5b1e0738a5d54d21b7173ceca4ad33fa492b03d8ac/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/e88cbb29a0d6de72bf322ce550431947b2c44758f4508df27a6f10748f0ebe06/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/bdb0e627a21f58fc411e801d2294f0fe514f6ad632690beee144fbc937ed6ec8/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/63099bed5b1b3ee6df81b86a68811bd8146788476eec755a56ad202540c26e4e/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/2a8bb9b23edea87881d86fa86aee98440bb5e931dd4c6ffbcf92569bce8a1b03/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a31773804693e801f2d1061806ab26c933e678fd0d1240ffa5d47b10b290209c/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/30e9e3bc7e9b5332d7f6e71813d03d83fe9ecb216aa0dd217524ded0a122f82b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/21fe740bb49d30b935589e086aee773306f1f13312433cb06d7180fd1bc45b23/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4632d4ef506b54fa02ead4a7d4263263fa0bd826e02c6f6b1c2f74aec7574b87/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 29 21:47:07.570063 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.569457    7082 manager.go:217] Machine: {Timestamp:2026-10-29 21:47:07.568890347 +0000 UTC m=+0.068932636 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3113015 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bbb418d50d8eb07d4891f292b865d SystemUUID:ec2bbb41-8d50-d8eb-07d4-891f292b865d BootID:d601cd98-4adb-425d-bb71-90259e43452f Filesystems:[{Device:/var/lib/kubelet/pods/baaab207-5534-4ef9-ac3a-9466ea5b900e/volumes/kubernetes.io~projected/kube-api-access-hkc5w DeviceMajor:0 DeviceMinor:356 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3259f5e5de34085e454c4fec63e4c26517488e625b618e564cdc7fc4f3ffe1d9/userdata/shm DeviceMajor:0 DeviceMinor:386 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3accc688-8a04-4462-9593-228a18548478/volumes/kubernetes.io~projected/kube-api-access-gnclr DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:280 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-433 DeviceMajor:0 DeviceMinor:433 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c20e4d88-bb47-4294-8242-2e71ca6d3f6c/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:465 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:301 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:383 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f683dbce-ebac-4d2c-ba04-aba3ac20792b/volumes/kubernetes.io~projected/kube-api-access-kck4b DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64227e95b780817a73780d1bd50fe0918a17f696017280aff7ff4d040aed8d6c/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/930a013f9bff036cccddc08345d999cc49d1c9bde3b77806a37c98ec3fc82c98/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3b64908a-fffb-4ec2-8697-51ea5db37f3b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3accc688-8a04-4462-9593-228a18548478/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae94d17c-b881-46e1-8063-e88c9dc53369/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:491 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d78fcfe0ae169d0d24484aec02afab6b7850f1079511868ac83f9d06b3b590b8/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bed90a2833cf42ce4a410e6c930d6ef7667ac810586d5f076818e1b5ecaad60f/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b5c45168936ec68706fa22cf5010febbd2abb5b13240dcbafa697326f78db6f/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:382 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd3c4499-8959-4616-842b-c556e7d23137/volumes/kubernetes.io~projected/kube-api-access-b6b42 DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3b64908a-fffb-4ec2-8697-51ea5db37f3b/volumes/kubernetes.io~projected/kube-api-access-bkhr5 DeviceMajor:0 DeviceMinor:244 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/baaab207-5534-4ef9-ac3a-9466ea5b900e/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:354 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4c2dd590-c639-42fe-8a02-4945b6403b3b/volumes/kubernetes.io~projected/kube-api-access-t5fvk DeviceMajor:0 DeviceMinor:355 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f64c76d-6e73-4cc0-9043-8f6235e0e281/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:478 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a09b130d-4249-4539-add0-e0aba69b752a/volumes/kubernetes.io~projected/kube-api-access-rgf24 DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1f41e102-fab5-4206-9502-fbf09515069a/volumes/kubernetes.io~projected/kube-api-access-8vrkn DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79bf4395-e3d6-4071-aedd-548c185fa39d/volumes/kubernetes.io~projected/kube-api-access-kmrsr DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66df3707a72a218383de890083a85d74eae68aee8ee9d173562b1d62817fae0a/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbf589c6e0dabaec0d066764594086a44f6bc30d6f4d74670ad6dca76ed2ad62/userdata/shm DeviceMajor:0 DeviceMinor:467 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/801f804c-187a-4232-af07-412a115a2fef/volumes/kubernetes.io~projected/kube-api-access-7rfd8 DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9999ce6f5d6e9d98d8a8f249aa37b8b9da08fa960a1f6bcf5c7b41c34d33db18/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/baaab207-5534-4ef9-ac3a-9466ea5b900e/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:353 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f64c76d-6e73-4cc0-9043-8f6235e0e281/volumes/kubernetes.io~projected/kube-api-access-sz6vz DeviceMajor:0 DeviceMinor:479 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d812d8da771abbedc07f0f5ed688a977dbdce8dae239dad6e12b15651d5b83b/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c893b43-cd05-45a0-9f7b-a2b5d72809c7/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-513 DeviceMajor:0 DeviceMinor:513 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:285 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c20e4d88-bb47-4294-8242-2e71ca6d3f6c/volumes/kubernetes.io~projected/kube-api-access-vcnrm DeviceMajor:0 DeviceMinor:466 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41b90bddf204131ae88a459287592030baa4a2335b039cc08b45c4d34dc96c8f/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:381 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f/volumes/kubernetes.io~projected/kube-api-access-drx2d DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:299 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e01d005eae6c14c03a9634425443b9564b8d5a3e28ded5ea1fc6347a08fec8ba/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/066041b1-7eba-4371-9470-101a7bf8fcfc/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32380f5c9fc4b9aa3b0dfab906c26f7d8179c9ff63f0b35e0533089efb12c27b/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:302 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd3c4499-8959-4616-842b-c556e7d23137/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:275 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b96159b5793d1be00bf8b618ba9cdf13625516b0c1b39e7ccea9a4794d29467/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/066041b1-7eba-4371-9470-101a7bf8fcfc/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-471 DeviceMajor:0 DeviceMinor:471 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f64c76d-6e73-4cc0-9043-8f6235e0e281/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:473 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef2a700a-5061-402a-ab9c-29a44204054c/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:486 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70b655ab16946107983c75a25d1a564c589aa3d4fc7751290f65f274ca5a4122/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:280 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b456e8e35c0f120681ba034e675eb1647a6833bc25a3e8850946e5f16a0f1370/userdata/shm DeviceMajor:0 DeviceMinor:493 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:293 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:300 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~projected/kube-api-access-vv8bm DeviceMajor:0 DeviceMinor:305 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60374fa76ee4647cf89872d467fec40c6a20477a19eccc4a213e21518f86c57a/userdata/shm DeviceMajor:0 DeviceMinor:480 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-507 DeviceMajor:0 DeviceMinor:507 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95e96df2fcf5249c4d02b0c62687b1782bb6e330bf154c7116adfc74f9765753/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ef2a700a-5061-402a-ab9c-29a44204054c/volumes/kubernetes.io~projected/kube-api-access-tbtl8 DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2deaa56200d86b0a245d5d869577bf52e61f8ae0b923ff824368b1099a340f41/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c893b43-cd05-45a0-9f7b-a2b5d72809c7/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:235 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:281 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:384 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd556ab388fe7dcbbd7f3786761067d0275bc14e79b20048da9eb0c29c52f28f/userdata/shm DeviceMajor:0 DeviceMinor:497 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5183b0be-c47a-4e35-a8cb-77991e6f7208/volumes/kubernetes.io~projected/kube-api-access-zpbcb DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a727e03aa88193020fda425d0065af63e966b5a3dfc59c8264358d9d7dc7c986/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/066041b1-7eba-4371-9470-101a7bf8fcfc/volumes/kubernetes.io~projected/kube-api-access-b7l79 DeviceMajor:0 DeviceMinor:245 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c46ed14e7584251dfb42b440bc26db3bec43422182a35183f02aec5e8ed6b66d/userdata/shm DeviceMajor:0 DeviceMinor:495 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79bf4395-e3d6-4071-aedd-548c185fa39d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b7ac84a5de67fab87859f8db39452d0ad41c49da9b34418888d7ff01e658423/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b7b55989043bc79ff2d50ed75a4b5de802ae3abe4a56324e8389bd9efd54e44/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0fadb982-a597-4489-ae16-3dc4fb5a7810/volumes/kubernetes.io~projected/kube-api-access-ht42j DeviceMajor:0 DeviceMinor:492 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/801f804c-187a-4232-af07-412a115a2fef/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-518 DeviceMajor:0 DeviceMinor:518 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/801f804c-187a-4232-af07-412a115a2fef/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dab249e6-5406-4ca6-87d5-d6fe41de7754/volumes/kubernetes.io~projected/kube-api-access-t9d2j DeviceMajor:0 DeviceMinor:385 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c20e4d88-bb47-4294-8242-2e71ca6d3f6c/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:461 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe53da9-f387-477c-8591-f5c37f42eaf5/volumes/kubernetes.io~projected/kube-api-access-v6m9n DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~projected/kube-api-access-klh9r DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe53da9-f387-477c-8591-f5c37f42eaf5/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2e1c568-867f-472f-9c39-47588d5f3fca/volumes/kubernetes.io~projected/kube-api-access-vshbg DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3239833dfc0a5983fdfc923651715e570c2cc3757bc1eaf4131ee67a1750d4d3/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c733459254465418be5838dcabb8ad525786cf2728369479663d2335fd3dee3/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4c2dd590-c639-42fe-8a02-4945b6403b3b/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:352 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/324d72a4-a641-4b09-99d6-d682b5e2c958/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1dd1bcbd88133555363bf351bf334c8198266dea4c32c7054d43127336b3f5cc/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c893b43-cd05-45a0-9f7b-a2b5d72809c7/volumes/kubernetes.io~projected/kube-api-access-ldwrg DeviceMajor:0 DeviceMinor:237 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3b64908a-fffb-4ec2-8697-51ea5db37f3b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:242 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3017c536-859c-4b7b-b59e-f1d5ee77e117/volumes/kubernetes.io~projected/kube-api-access-zvtbk DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b96d72b2-bc45-4962-b529-f08155ad0215/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:292 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/baaab207-5534-4ef9-ac3a-9466ea5b900e/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:347 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-499 DeviceMajor:0 DeviceMinor:499 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-505 DeviceMajor:0 DeviceMinor:505 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe53da9-f387-477c-8591-f5c37f42eaf5/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1b5c45168936ec6 MacAddress:f6:2c:7d:f8:f7:79 Speed:10000 Mtu:8901} {Name:2b7ac84a5de67fa MacAddress:7e:51:dc:53:7c:fd Speed:10000 Mtu:8901} {Name:32380f5c9fc4b9a MacAddress:82:9b:d1:db:6d:d8 Speed:10000 Mtu:8901} {Name:3239833dfc0a598 MacAddress:b2:7b:ad:c3:dc:66 Speed:10000 Mtu:8901} {Name:3259f5e5de34085 MacAddress:46:8d:33:b4:e8:ca Speed:10000 Mtu:8901} {Name:3d812d8da771abb MacAddress:1e:97:02:42:be:bb Speed:10000 Mtu:8901} {Name:4b7b55989043bc7 MacAddress:92:6f:36:ea:3b:cc Speed:10000 Mtu:8901} {Name:5b96159b5793d1b MacAddress:32:60:91:5c:0d:aa Speed:10000 Mtu:8901} {Name:60374fa76ee4647 MacAddress:12:2c:00:57:77:07 Speed:10000 Mtu:8901} {Name:66df3707a72a218 MacAddress:26:f4:a0:13:d0:1d Speed:10000 Mtu:8901} {Name:9999ce6f5d6e9d9 MacAddress:82:59:dd:91:aa:f0 Speed:10000 Mtu:8901} {Name:b456e8e35c0f120 MacAddress:06:5a:50:dc:de:0a Speed:10000 Mtu:8901} {Name:bed90a2833cf42c MacAddress:2e:bc:1f:a6:83:f0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d2:42:99:0f Speed:0 Mtu:9001} {Name:br-int MacAddress:d2:2d:f2:3e:97:7e Speed:0 Mtu:8901} {Name:c46ed14e7584251 MacAddress:0a:69:16:60:77:b8 Speed:10000 Mtu:8901} {Name:cd556ab388fe7dc MacAddress:d6:a2:84:ed:c0:0e Speed:10000 Mtu:8901} {Name:dbf589c6e0dabae MacAddress:62:01:0a:13:b5:56 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d2:42:99:0f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b6:a5:06:2a:c7:34 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fa:b4:91:d2:34:50 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 21:47:07.570063 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.570066    7082 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 21:47:07.570534 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.570140    7082 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 21:47:07.570588 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.570548    7082 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 21:47:07.570793 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.570591    7082 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-38.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 21:47:07.570887 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.570797    7082 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 21:47:07.570887 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.570809    7082 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 21:47:07.570887 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.570825    7082 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 21:47:07.570887 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.570852    7082 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 21:47:07.571034 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.571027    7082 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 21:47:07.571254 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.571239    7082 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 21:47:07.571393 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.571382    7082 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 21:47:07.571455 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.571437    7082 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 21:47:07.571504 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.571459    7082 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 21:47:07.571504 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.571474    7082 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 21:47:07.571504 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.571487    7082 kubelet.go:405] "Adding apiserver pod source"
Oct 29 21:47:07.571504 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.571499    7082 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 21:47:07.572314 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.572297    7082 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 21:47:07.572314 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.572313    7082 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 21:47:07.572943 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.572924    7082 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 21:47:07.575858 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.575841    7082 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 21:47:07.575858 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.575857    7082 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 21:47:07.575971 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.575961    7082 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 29 21:47:07.576210 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576197    7082 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 21:47:07.576284 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576229    7082 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 21:47:07.576346 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576335    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 21:47:07.576376 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576355    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 21:47:07.576376 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576369    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 21:47:07.576426 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576377    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 21:47:07.576426 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576384    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 21:47:07.576426 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576391    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 21:47:07.576426 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576398    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 21:47:07.576426 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576403    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 21:47:07.576426 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576411    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 21:47:07.576426 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576417    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 21:47:07.576614 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576430    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 21:47:07.576614 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576442    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 21:47:07.576614 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576503    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 21:47:07.576614 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.576510    7082 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 21:47:07.580495 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.580473    7082 server.go:1263] "Started kubelet"
Oct 29 21:47:07.580613 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.580543    7082 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 21:47:07.580726 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.580663    7082 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 21:47:07.580808 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.580756    7082 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 21:47:07.581128 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.581107    7082 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 21:47:07.581190 ip-10-0-136-38 systemd[1]: Started Kubernetes Kubelet.
Oct 29 21:47:07.582837 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.582223    7082 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 21:47:07.588138 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.588110    7082 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 21:47:07.588570 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.588533    7082 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 21:47:07.588976 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.588963    7082 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 21:47:07.588976 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.588977    7082 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 21:47:07.589329 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.589295    7082 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:40:58 +0000 UTC" deadline="2028-05-03 05:38:14.777633618 +0000 UTC"
Oct 29 21:47:07.589401 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.589330    7082 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13231h51m7.188307534s"
Oct 29 21:47:07.589401 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.589381    7082 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 21:47:07.589561 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:07.589547    7082 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-38.ec2.internal\" not found"
Oct 29 21:47:07.590760 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.590735    7082 factory.go:55] Registering systemd factory
Oct 29 21:47:07.590760 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.590761    7082 factory.go:223] Registration of the systemd container factory successfully
Oct 29 21:47:07.591063 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.591043    7082 factory.go:153] Registering CRI-O factory
Oct 29 21:47:07.591142 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.591067    7082 factory.go:223] Registration of the crio container factory successfully
Oct 29 21:47:07.591142 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.591131    7082 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 21:47:07.591251 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.591152    7082 factory.go:103] Registering Raw factory
Oct 29 21:47:07.591251 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.591163    7082 manager.go:1199] Started watching for new ooms in manager
Oct 29 21:47:07.591505 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.591490    7082 manager.go:319] Starting recovery of all containers
Oct 29 21:47:07.593678 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593576    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" volumeName="kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-tls-assets" seLinuxMountContext=""
Oct 29 21:47:07.593854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593683    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" volumeName="kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-service-ca" seLinuxMountContext=""
Oct 29 21:47:07.593854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593699    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96d72b2-bc45-4962-b529-f08155ad0215" volumeName="kubernetes.io/projected/b96d72b2-bc45-4962-b529-f08155ad0215-kube-api-access-vv8bm" seLinuxMountContext=""
Oct 29 21:47:07.593854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593712    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" volumeName="kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-out" seLinuxMountContext=""
Oct 29 21:47:07.593854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593724    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" volumeName="kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 29 21:47:07.593854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593737    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a09b130d-4249-4539-add0-e0aba69b752a" volumeName="kubernetes.io/empty-dir/a09b130d-4249-4539-add0-e0aba69b752a-tmp-dir" seLinuxMountContext=""
Oct 29 21:47:07.593854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593750    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" volumeName="kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-oauth-config" seLinuxMountContext=""
Oct 29 21:47:07.593854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593762    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" volumeName="kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-config" seLinuxMountContext=""
Oct 29 21:47:07.593854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593800    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f64c76d-6e73-4cc0-9043-8f6235e0e281" volumeName="kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-config" seLinuxMountContext=""
Oct 29 21:47:07.593854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593812    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801f804c-187a-4232-af07-412a115a2fef" volumeName="kubernetes.io/projected/801f804c-187a-4232-af07-412a115a2fef-kube-api-access-7rfd8" seLinuxMountContext=""
Oct 29 21:47:07.593854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593824    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b64908a-fffb-4ec2-8697-51ea5db37f3b" volumeName="kubernetes.io/empty-dir/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-textfile" seLinuxMountContext=""
Oct 29 21:47:07.593854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593835    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" volumeName="kubernetes.io/empty-dir/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-tmp" seLinuxMountContext=""
Oct 29 21:47:07.593854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593849    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab249e6-5406-4ca6-87d5-d6fe41de7754" volumeName="kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593860    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801f804c-187a-4232-af07-412a115a2fef" volumeName="kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-ovnkube-script-lib" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593877    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c893b43-cd05-45a0-9f7b-a2b5d72809c7" volumeName="kubernetes.io/projected/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-kube-api-access-ldwrg" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593889    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2e1c568-867f-472f-9c39-47588d5f3fca" volumeName="kubernetes.io/configmap/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-daemon-config" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593902    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96d72b2-bc45-4962-b529-f08155ad0215" volumeName="kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593914    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" volumeName="kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-oauth-serving-cert" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593927    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baaab207-5534-4ef9-ac3a-9466ea5b900e" volumeName="kubernetes.io/empty-dir/baaab207-5534-4ef9-ac3a-9466ea5b900e-audit-log" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593939    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe53da9-f387-477c-8591-f5c37f42eaf5" volumeName="kubernetes.io/projected/bbe53da9-f387-477c-8591-f5c37f42eaf5-kube-api-access-v6m9n" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593951    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="066041b1-7eba-4371-9470-101a7bf8fcfc" volumeName="kubernetes.io/projected/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-api-access-b7l79" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593963    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801f804c-187a-4232-af07-412a115a2fef" volumeName="kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-ovnkube-config" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593974    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae94d17c-b881-46e1-8063-e88c9dc53369" volumeName="kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.593987    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe53da9-f387-477c-8591-f5c37f42eaf5" volumeName="kubernetes.io/empty-dir/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-tuned" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594022    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab249e6-5406-4ca6-87d5-d6fe41de7754" volumeName="kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-telemeter-client-tls" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594035    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="066041b1-7eba-4371-9470-101a7bf8fcfc" volumeName="kubernetes.io/empty-dir/066041b1-7eba-4371-9470-101a7bf8fcfc-volume-directive-shadow" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594047    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" volumeName="kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-oauth-config" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594062    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab249e6-5406-4ca6-87d5-d6fe41de7754" volumeName="kubernetes.io/projected/dab249e6-5406-4ca6-87d5-d6fe41de7754-kube-api-access-t9d2j" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594082    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" volumeName="kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-volume" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594096    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3accc688-8a04-4462-9593-228a18548478" volumeName="kubernetes.io/secret/3accc688-8a04-4462-9593-228a18548478-metrics-tls" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594110    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c893b43-cd05-45a0-9f7b-a2b5d72809c7" volumeName="kubernetes.io/secret/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594127    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a09b130d-4249-4539-add0-e0aba69b752a" volumeName="kubernetes.io/projected/a09b130d-4249-4539-add0-e0aba69b752a-kube-api-access-rgf24" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594139    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab249e6-5406-4ca6-87d5-d6fe41de7754" volumeName="kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-federate-client-tls" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594155    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="066041b1-7eba-4371-9470-101a7bf8fcfc" volumeName="kubernetes.io/configmap/066041b1-7eba-4371-9470-101a7bf8fcfc-metrics-client-ca" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594167    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" volumeName="kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-config" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594180    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96d72b2-bc45-4962-b529-f08155ad0215" volumeName="kubernetes.io/empty-dir/b96d72b2-bc45-4962-b529-f08155ad0215-tmp" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594196    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" volumeName="kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-trusted-ca-bundle" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594224    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab249e6-5406-4ca6-87d5-d6fe41de7754" volumeName="kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594238    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f64c76d-6e73-4cc0-9043-8f6235e0e281" volumeName="kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-oauth-config" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594251    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b64908a-fffb-4ec2-8697-51ea5db37f3b" volumeName="kubernetes.io/configmap/3b64908a-fffb-4ec2-8697-51ea5db37f3b-metrics-client-ca" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594268    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5183b0be-c47a-4e35-a8cb-77991e6f7208" volumeName="kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594283    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79bf4395-e3d6-4071-aedd-548c185fa39d" volumeName="kubernetes.io/empty-dir/79bf4395-e3d6-4071-aedd-548c185fa39d-data-volume" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594297    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" volumeName="kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594309    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b64908a-fffb-4ec2-8697-51ea5db37f3b" volumeName="kubernetes.io/secret/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594322    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96d72b2-bc45-4962-b529-f08155ad0215" volumeName="kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594334    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" volumeName="kubernetes.io/empty-dir/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-tmp" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594347    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f683dbce-ebac-4d2c-ba04-aba3ac20792b" volumeName="kubernetes.io/projected/f683dbce-ebac-4d2c-ba04-aba3ac20792b-kube-api-access-kck4b" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594359    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="066041b1-7eba-4371-9470-101a7bf8fcfc" volumeName="kubernetes.io/configmap/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594373    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b64908a-fffb-4ec2-8697-51ea5db37f3b" volumeName="kubernetes.io/projected/3b64908a-fffb-4ec2-8697-51ea5db37f3b-kube-api-access-bkhr5" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594387    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c2dd590-c639-42fe-8a02-4945b6403b3b" volumeName="kubernetes.io/secret/4c2dd590-c639-42fe-8a02-4945b6403b3b-monitoring-plugin-cert" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594416    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5183b0be-c47a-4e35-a8cb-77991e6f7208" volumeName="kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-cni-binary-copy" seLinuxMountContext=""
Oct 29 21:47:07.594397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594428    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c893b43-cd05-45a0-9f7b-a2b5d72809c7" volumeName="kubernetes.io/secret/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594440    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baaab207-5534-4ef9-ac3a-9466ea5b900e" volumeName="kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-secret-metrics-server-tls" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594453    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f64c76d-6e73-4cc0-9043-8f6235e0e281" volumeName="kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-serving-cert" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594466    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801f804c-187a-4232-af07-412a115a2fef" volumeName="kubernetes.io/secret/801f804c-187a-4232-af07-412a115a2fef-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594479    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3017c536-859c-4b7b-b59e-f1d5ee77e117" volumeName="kubernetes.io/configmap/3017c536-859c-4b7b-b59e-f1d5ee77e117-iptables-alerter-script" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594503    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" volumeName="kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594534    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3accc688-8a04-4462-9593-228a18548478" volumeName="kubernetes.io/configmap/3accc688-8a04-4462-9593-228a18548478-config-volume" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594547    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baaab207-5534-4ef9-ac3a-9466ea5b900e" volumeName="kubernetes.io/configmap/baaab207-5534-4ef9-ac3a-9466ea5b900e-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594560    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baaab207-5534-4ef9-ac3a-9466ea5b900e" volumeName="kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594590    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fadb982-a597-4489-ae16-3dc4fb5a7810" volumeName="kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594604    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2e1c568-867f-472f-9c39-47588d5f3fca" volumeName="kubernetes.io/projected/c2e1c568-867f-472f-9c39-47588d5f3fca-kube-api-access-vshbg" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594617    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f64c76d-6e73-4cc0-9043-8f6235e0e281" volumeName="kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-service-ca" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594648    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" volumeName="kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-serving-cert" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594662    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab249e6-5406-4ca6-87d5-d6fe41de7754" volumeName="kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-metrics-client-ca" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594676    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f64c76d-6e73-4cc0-9043-8f6235e0e281" volumeName="kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-oauth-serving-cert" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594691    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" volumeName="kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-cluster-tls-config" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594704    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="324d72a4-a641-4b09-99d6-d682b5e2c958" volumeName="kubernetes.io/secret/324d72a4-a641-4b09-99d6-d682b5e2c958-agent-certs" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594716    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5183b0be-c47a-4e35-a8cb-77991e6f7208" volumeName="kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594729    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b64908a-fffb-4ec2-8697-51ea5db37f3b" volumeName="kubernetes.io/secret/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-tls" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594742    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef2a700a-5061-402a-ab9c-29a44204054c" volumeName="kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594753    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b64908a-fffb-4ec2-8697-51ea5db37f3b" volumeName="kubernetes.io/configmap/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594767    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baaab207-5534-4ef9-ac3a-9466ea5b900e" volumeName="kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-client-ca-bundle" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594779    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3017c536-859c-4b7b-b59e-f1d5ee77e117" volumeName="kubernetes.io/projected/3017c536-859c-4b7b-b59e-f1d5ee77e117-kube-api-access-zvtbk" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594792    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" volumeName="kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-main-db" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594804    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5183b0be-c47a-4e35-a8cb-77991e6f7208" volumeName="kubernetes.io/projected/5183b0be-c47a-4e35-a8cb-77991e6f7208-kube-api-access-zpbcb" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594816    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c893b43-cd05-45a0-9f7b-a2b5d72809c7" volumeName="kubernetes.io/configmap/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-metrics-client-ca" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594829    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe53da9-f387-477c-8591-f5c37f42eaf5" volumeName="kubernetes.io/empty-dir/bbe53da9-f387-477c-8591-f5c37f42eaf5-tmp" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594842    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd3c4499-8959-4616-842b-c556e7d23137" volumeName="kubernetes.io/projected/cd3c4499-8959-4616-842b-c556e7d23137-kube-api-access-b6b42" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594855    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" volumeName="kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594867    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3accc688-8a04-4462-9593-228a18548478" volumeName="kubernetes.io/empty-dir/3accc688-8a04-4462-9593-228a18548478-tmp-dir" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594879    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79bf4395-e3d6-4071-aedd-548c185fa39d" volumeName="kubernetes.io/configmap/79bf4395-e3d6-4071-aedd-548c185fa39d-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594895    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96d72b2-bc45-4962-b529-f08155ad0215" volumeName="kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.594908    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baaab207-5534-4ef9-ac3a-9466ea5b900e" volumeName="kubernetes.io/configmap/baaab207-5534-4ef9-ac3a-9466ea5b900e-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595216    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab249e6-5406-4ca6-87d5-d6fe41de7754" volumeName="kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-secret-telemeter-client" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595240    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" volumeName="kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-metrics-client-ca" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595254    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" volumeName="kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-kube-api-access-klh9r" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595268    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" volumeName="kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-web-config" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595280    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" volumeName="kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595295    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" volumeName="kubernetes.io/projected/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-kube-api-access-drx2d" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595309    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baaab207-5534-4ef9-ac3a-9466ea5b900e" volumeName="kubernetes.io/projected/baaab207-5534-4ef9-ac3a-9466ea5b900e-kube-api-access-hkc5w" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595322    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" volumeName="kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-service-ca" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595353    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd3c4499-8959-4616-842b-c556e7d23137" volumeName="kubernetes.io/secret/cd3c4499-8959-4616-842b-c556e7d23137-cert" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595368    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f41e102-fab5-4206-9502-fbf09515069a" volumeName="kubernetes.io/configmap/1f41e102-fab5-4206-9502-fbf09515069a-serviceca" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595382    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="324d72a4-a641-4b09-99d6-d682b5e2c958" volumeName="kubernetes.io/configmap/324d72a4-a641-4b09-99d6-d682b5e2c958-konnectivity-ca" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595394    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79bf4395-e3d6-4071-aedd-548c185fa39d" volumeName="kubernetes.io/projected/79bf4395-e3d6-4071-aedd-548c185fa39d-kube-api-access-kmrsr" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595416    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801f804c-187a-4232-af07-412a115a2fef" volumeName="kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-env-overrides" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595429    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" volumeName="kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-oauth-serving-cert" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595499    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" volumeName="kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-serving-cert" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595515    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96d72b2-bc45-4962-b529-f08155ad0215" volumeName="kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595530    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="066041b1-7eba-4371-9470-101a7bf8fcfc" volumeName="kubernetes.io/secret/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-tls" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595543    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c2dd590-c639-42fe-8a02-4945b6403b3b" volumeName="kubernetes.io/projected/4c2dd590-c639-42fe-8a02-4945b6403b3b-kube-api-access-t5fvk" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595556    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79bf4395-e3d6-4071-aedd-548c185fa39d" volumeName="kubernetes.io/secret/79bf4395-e3d6-4071-aedd-548c185fa39d-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595569    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab249e6-5406-4ca6-87d5-d6fe41de7754" volumeName="kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595581    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f41e102-fab5-4206-9502-fbf09515069a" volumeName="kubernetes.io/projected/1f41e102-fab5-4206-9502-fbf09515069a-kube-api-access-8vrkn" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595594    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f64c76d-6e73-4cc0-9043-8f6235e0e281" volumeName="kubernetes.io/empty-dir/2f64c76d-6e73-4cc0-9043-8f6235e0e281-tmp" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595607    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2e1c568-867f-472f-9c39-47588d5f3fca" volumeName="kubernetes.io/configmap/c2e1c568-867f-472f-9c39-47588d5f3fca-cni-binary-copy" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595619    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="066041b1-7eba-4371-9470-101a7bf8fcfc" volumeName="kubernetes.io/secret/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595651    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96d72b2-bc45-4962-b529-f08155ad0215" volumeName="kubernetes.io/configmap/b96d72b2-bc45-4962-b529-f08155ad0215-metrics-client-ca" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595664    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96d72b2-bc45-4962-b529-f08155ad0215" volumeName="kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-grpc-tls" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595678    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b96d72b2-bc45-4962-b529-f08155ad0215" volumeName="kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595689    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef2a700a-5061-402a-ab9c-29a44204054c" volumeName="kubernetes.io/projected/ef2a700a-5061-402a-ab9c-29a44204054c-kube-api-access-tbtl8" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595723    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f64c76d-6e73-4cc0-9043-8f6235e0e281" volumeName="kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-trusted-ca-bundle" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595736    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f64c76d-6e73-4cc0-9043-8f6235e0e281" volumeName="kubernetes.io/projected/2f64c76d-6e73-4cc0-9043-8f6235e0e281-kube-api-access-sz6vz" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595755    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3accc688-8a04-4462-9593-228a18548478" volumeName="kubernetes.io/projected/3accc688-8a04-4462-9593-228a18548478-kube-api-access-gnclr" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595786    7082 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" volumeName="kubernetes.io/projected/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-kube-api-access-vcnrm" seLinuxMountContext=""
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595799    7082 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 21:47:07.596453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.595807    7082 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 21:47:07.620944 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.620896    7082 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 21:47:07.622718 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.622691    7082 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 21:47:07.622718 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.622714    7082 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 21:47:07.622919 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.622736    7082 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 21:47:07.622919 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:07.622784    7082 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 21:47:07.641357 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.641324    7082 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:47:07.641567 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.641436    7082 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 29 21:47:07.643568 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.643544    7082 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:47:07.655817 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.655777    7082 generic.go:356] "Generic (PLEG): container finished" podID="39a608f7d1c53c952b07789e6ea659e9" containerID="499d23babcc83b7dcc144bbcc9f1e5a27350c8da01f29a9799c6e606b4d61010" exitCode=0
Oct 29 21:47:07.665677 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.665647    7082 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:47:07.666389 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.666314    7082 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:47:07.681496 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.681407    7082 generic.go:356] "Generic (PLEG): container finished" podID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerID="853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26" exitCode=0
Oct 29 21:47:07.687842 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.687793    7082 generic.go:356] "Generic (PLEG): container finished" podID="5183b0be-c47a-4e35-a8cb-77991e6f7208" containerID="beb1209d08fa4d6dd6029800dc6cd4f31cf27bfbc3216b7a4b4d114794217a7c" exitCode=0
Oct 29 21:47:07.687842 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.687824    7082 generic.go:356] "Generic (PLEG): container finished" podID="5183b0be-c47a-4e35-a8cb-77991e6f7208" containerID="308bc48fe717d9352916673b8ab574dbe54e44835836425fecd63d66afdf8fb6" exitCode=0
Oct 29 21:47:07.687842 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.687831    7082 generic.go:356] "Generic (PLEG): container finished" podID="5183b0be-c47a-4e35-a8cb-77991e6f7208" containerID="02fe1cc540f7281ac0e2aafbbdcea73a4042bf472b07c24ad527061378e60df8" exitCode=0
Oct 29 21:47:07.687842 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.687836    7082 generic.go:356] "Generic (PLEG): container finished" podID="5183b0be-c47a-4e35-a8cb-77991e6f7208" containerID="a9b10383bc398569e9cbf1f28287ea8c2e1db59f64ec81c568fa1c925cb1c774" exitCode=0
Oct 29 21:47:07.687842 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.687841    7082 generic.go:356] "Generic (PLEG): container finished" podID="5183b0be-c47a-4e35-a8cb-77991e6f7208" containerID="529a352b49a16fc1999791c8190b7cfa9cb16b8e8d33b2da1d1db4965960d4bf" exitCode=0
Oct 29 21:47:07.687842 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.687846    7082 generic.go:356] "Generic (PLEG): container finished" podID="5183b0be-c47a-4e35-a8cb-77991e6f7208" containerID="e79587e333bee33eec85d67583efa01672a06204efa0c5fee0b048ced5d32a9c" exitCode=0
Oct 29 21:47:07.698136 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.698094    7082 generic.go:356] "Generic (PLEG): container finished" podID="3b64908a-fffb-4ec2-8697-51ea5db37f3b" containerID="8c0116cd936f5877e4cb58a85f0cf1a63a8423bc9fcb1aac5815e01232ac54d8" exitCode=0
Oct 29 21:47:07.724358 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:07.724302    7082 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 29 21:47:07.729471 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.729443    7082 manager.go:324] Recovery completed
Oct 29 21:47:07.760642 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.760604    7082 cpu_manager.go:226] "Starting" policy="none"
Oct 29 21:47:07.760642 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.760641    7082 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 21:47:07.760856 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.760675    7082 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 21:47:07.761397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.760899    7082 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 29 21:47:07.761433 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.761402    7082 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 29 21:47:07.761433 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.761428    7082 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 29 21:47:07.761492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.761437    7082 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 29 21:47:07.761492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.761446    7082 policy_none.go:50] "Start"
Oct 29 21:47:07.761492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.761461    7082 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 21:47:07.761492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.761477    7082 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 21:47:07.762394 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.762377    7082 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 29 21:47:07.762432 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.762395    7082 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 29 21:47:07.762432 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.762404    7082 policy_none.go:44] "Start"
Oct 29 21:47:07.767746 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.767703    7082 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 21:47:07.772874 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.772841    7082 manager.go:341] "Starting Device Plugin manager"
Oct 29 21:47:07.772874 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:07.772877    7082 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 21:47:07.773113 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.772891    7082 server.go:87] "Starting device plugin registration server"
Oct 29 21:47:07.773223 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.773209    7082 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 21:47:07.773286 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.773223    7082 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 21:47:07.773335 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.773303    7082 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 21:47:07.773446 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.773405    7082 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 21:47:07.773570 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.773453    7082 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 21:47:07.773570 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.773461    7082 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 21:47:07.773703 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.773522    7082 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T21:47:07.773441338Z","UUID":"7f11c8cf-a730-4796-92e0-6e620ca19dfc","Handler":null,"Name":"","Endpoint":""}
Oct 29 21:47:07.775169 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.775146    7082 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 21:47:07.775281 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.775189    7082 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 21:47:07.776131 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:07.776109    7082 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 21:47:07.874823 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.874782    7082 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:47:07.877277 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.877254    7082 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:47:07.877395 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.877291    7082 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:47:07.877395 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.877301    7082 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-38.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:47:07.877395 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.877349    7082 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-38.ec2.internal"
Oct 29 21:47:07.914300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.914261    7082 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-38.ec2.internal"
Oct 29 21:47:07.914443 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.914364    7082 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-38.ec2.internal"
Oct 29 21:47:07.925179 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.925103    7082 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"]
Oct 29 21:47:07.925883 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.925845    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal" event={"ID":"39cf47f8a2a071911c6009d122ac3420","Type":"ContainerStarted","Data":"710ddbcfc63f91fae46a6a3e95cb1b47fbda598be2d45ab724c451183516fd9a"}
Oct 29 21:47:07.925945 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.925892    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal" event={"ID":"39cf47f8a2a071911c6009d122ac3420","Type":"ContainerStarted","Data":"95e96df2fcf5249c4d02b0c62687b1782bb6e330bf154c7116adfc74f9765753"}
Oct 29 21:47:07.925945 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.925899    7082 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:07.926014 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.925906    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal" event={"ID":"39a608f7d1c53c952b07789e6ea659e9","Type":"ContainerStarted","Data":"1b36b2348f6eed576a2e5a5bb9d950b33de393beb0004c441d829efc84602b86"}
Oct 29 21:47:07.926060 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.926037    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal" event={"ID":"39a608f7d1c53c952b07789e6ea659e9","Type":"ContainerDied","Data":"499d23babcc83b7dcc144bbcc9f1e5a27350c8da01f29a9799c6e606b4d61010"}
Oct 29 21:47:07.926090 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.926075    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal" event={"ID":"39a608f7d1c53c952b07789e6ea659e9","Type":"ContainerStarted","Data":"64227e95b780817a73780d1bd50fe0918a17f696017280aff7ff4d040aed8d6c"}
Oct 29 21:47:07.926201 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.926188    7082 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:07.962662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.962553    7082 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:47:07.962662 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:07.962652    7082 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-38.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:07.963253 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:07.963216    7082 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:47:07.963367 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:07.963263    7082 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:08.043691 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.043657    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39a608f7d1c53c952b07789e6ea659e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"39a608f7d1c53c952b07789e6ea659e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:08.043691 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.043693    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39a608f7d1c53c952b07789e6ea659e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"39a608f7d1c53c952b07789e6ea659e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:08.043943 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.043712    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39cf47f8a2a071911c6009d122ac3420-config\") pod \"kube-apiserver-proxy-ip-10-0-136-38.ec2.internal\" (UID: \"39cf47f8a2a071911c6009d122ac3420\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:08.144290 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.144244    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39a608f7d1c53c952b07789e6ea659e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"39a608f7d1c53c952b07789e6ea659e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:08.144290 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.144287    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39cf47f8a2a071911c6009d122ac3420-config\") pod \"kube-apiserver-proxy-ip-10-0-136-38.ec2.internal\" (UID: \"39cf47f8a2a071911c6009d122ac3420\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:08.144524 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.144352    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/39cf47f8a2a071911c6009d122ac3420-config\") pod \"kube-apiserver-proxy-ip-10-0-136-38.ec2.internal\" (UID: \"39cf47f8a2a071911c6009d122ac3420\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:08.144524 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.144365    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39a608f7d1c53c952b07789e6ea659e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"39a608f7d1c53c952b07789e6ea659e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:08.144524 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.144413    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39a608f7d1c53c952b07789e6ea659e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"39a608f7d1c53c952b07789e6ea659e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:08.144524 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.144467    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39a608f7d1c53c952b07789e6ea659e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" (UID: \"39a608f7d1c53c952b07789e6ea659e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:08.572221 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.572179    7082 apiserver.go:52] "Watching apiserver"
Oct 29 21:47:08.623649 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.623601    7082 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 21:47:08.624448 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.624417    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-dmgg8","openshift-monitoring/openshift-state-metrics-7bd7489957-js76b","openshift-multus/multus-additional-cni-plugins-gd2nd","openshift-multus/multus-n4nsq","openshift-multus/network-metrics-daemon-4zxrn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-38.ec2.internal","openshift-cluster-node-tuning-operator/tuned-frxl2","openshift-console/console-968f5bf44-dhgp8","openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2","kube-system/global-pull-secret-syncer-vth7d","openshift-monitoring/metrics-server-f7fc46b4-vvxlv","openshift-monitoring/node-exporter-wtbq7","openshift-network-diagnostics/network-check-target-n8m99","openshift-network-operator/iptables-alerter-l7z5g","openshift-ingress-canary/ingress-canary-zv5gd","openshift-insights/insights-runtime-extractor-2867l","openshift-monitoring/telemeter-client-54969c98b4-7zdlx","openshift-console/console-779475d747-twkqw","openshift-ovn-kubernetes/ovnkube-node-pzj9l","openshift-dns/dns-default-245p6","openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4","openshift-dns/node-resolver-8jkmb","kube-system/konnectivity-agent-vwk66","openshift-monitoring/thanos-querier-f96f5cd85-t7ndb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g","openshift-console/console-6c7d74bc46-5bb2n","openshift-monitoring/alertmanager-main-0"]
Oct 29 21:47:08.640795 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.640757    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 21:47:08.640795 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.640783    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 21:47:08.641029 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.640829    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 21:47:08.641029 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.640853    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 21:47:08.641029 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.640929    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 21:47:08.641029 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.641017    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 21:47:08.641230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.641120    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 21:47:08.641230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.641148    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:47:08.641230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.641149    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 21:47:08.641230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.641162    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 21:47:08.641230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.641202    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 21:47:08.641479 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.640777    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 21:47:08.654300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654265    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 21:47:08.654300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654281    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 21:47:08.654498 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654363    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 21:47:08.654498 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654420    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 21:47:08.654644 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654500    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 21:47:08.654644 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654505    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 21:47:08.654644 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654532    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 21:47:08.654644 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654548    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 21:47:08.654644 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654600    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 29 21:47:08.654840 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654684    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 21:47:08.654840 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654742    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 21:47:08.654840 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654764    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 21:47:08.654840 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654769    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 21:47:08.654840 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654783    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 21:47:08.654840 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654784    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 21:47:08.655041 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654848    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 21:47:08.655041 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654866    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 29 21:47:08.655041 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654899    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 21:47:08.655041 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.654925    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 21:47:08.655041 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.655016    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 21:47:08.655604 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.655554    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 21:47:08.655715 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.655589    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 21:47:08.655715 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.655707    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 29 21:47:08.655855 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.655716    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 21:47:08.655855 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.655761    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 29 21:47:08.655855 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.655776    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 21:47:08.655855 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.655849    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 29 21:47:08.656782 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.656767    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 21:47:08.656939 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.656928    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 29 21:47:08.692839 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.692807    7082 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 21:47:08.698661 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.698622    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 29 21:47:08.700345 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.700328    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 21:47:08.700781 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.700755    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 29 21:47:08.700894 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.700781    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 29 21:47:08.700894 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.700758    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 29 21:47:08.700894 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.700758    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 29 21:47:08.701070 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.700965    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 21:47:08.701070 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701002    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 29 21:47:08.701070 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701005    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-znmmx\""
Oct 29 21:47:08.701239 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701220    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 21:47:08.701292 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701279    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-p5fxq\""
Oct 29 21:47:08.701334 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701306    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vdqjz\""
Oct 29 21:47:08.701334 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701319    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 21:47:08.701334 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701324    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w9rdw\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701462    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701466    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6g5r7\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701480    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701474    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pjw5g\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701512    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701522    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701525    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701555    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701562    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701589    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701597    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701601    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701609    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701655    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ghrw4\""
Oct 29 21:47:08.701696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701675    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701832    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701859    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mlr8g\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701893    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9qchq\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701914    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lssrn\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701921    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701933    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6fjlx\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701946    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701980    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701936    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-wqq76\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701994    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.701992    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.702012    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8ld6j\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.702040    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.702056    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsffp\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.702068    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zcf2k\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.702085    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.702111    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9d1tb9gv0p1rk\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.702129    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fzqww\""
Oct 29 21:47:08.702288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.702154    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 29 21:47:08.702845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.702364    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 21:47:08.702845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.702459    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 29 21:47:08.702903 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.702888    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 29 21:47:08.704092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.704074    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 21:47:08.704092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.704087    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l6fj9\""
Oct 29 21:47:08.704230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.704091    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hzxgj\""
Oct 29 21:47:08.704230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.704110    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-296ss7dcv4g3a\""
Oct 29 21:47:08.704230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.704127    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 29 21:47:08.704230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.704137    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4sldf\""
Oct 29 21:47:08.704230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.704144    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6jh7q\""
Oct 29 21:47:08.704230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.704127    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6chnm\""
Oct 29 21:47:08.704230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.704152    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ckcdt\""
Oct 29 21:47:08.704230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.704164    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5fns5\""
Oct 29 21:47:08.704230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.704166    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7dqcg\""
Oct 29 21:47:08.710463 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.710432    7082 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:08.736351 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.736317    7082 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:47:08.736529 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:08.736385    7082 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal"
Oct 29 21:47:08.740242 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.740209    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 29 21:47:08.740422 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.740297    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:47:08.740480 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.740429    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 29 21:47:08.740997 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.740976    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 29 21:47:08.748036 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.747999    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-hostroot\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.748036 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748042    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/324d72a4-a641-4b09-99d6-d682b5e2c958-konnectivity-ca\") pod \"konnectivity-agent-vwk66\" (UID: \"324d72a4-a641-4b09-99d6-d682b5e2c958\") " pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:47:08.748257 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748081    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwrg\" (UniqueName: \"kubernetes.io/projected/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-kube-api-access-ldwrg\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:47:08.748257 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748234    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-trusted-ca-bundle\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.748392 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748270    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-oauth-serving-cert\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.748392 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748302    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6m9n\" (UniqueName: \"kubernetes.io/projected/bbe53da9-f387-477c-8591-f5c37f42eaf5-kube-api-access-v6m9n\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.748392 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748328    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.748392 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748352    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-tmp\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.748392 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748372    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/324d72a4-a641-4b09-99d6-d682b5e2c958-konnectivity-ca\") pod \"konnectivity-agent-vwk66\" (UID: \"324d72a4-a641-4b09-99d6-d682b5e2c958\") " pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:47:08.748392 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748382    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.748687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748417    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rfd8\" (UniqueName: \"kubernetes.io/projected/801f804c-187a-4232-af07-412a115a2fef-kube-api-access-7rfd8\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.748687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748449    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-tmp\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.748687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748525    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-k8s-cni-cncf-io\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.748687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748558    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-cni-multus\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.748687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748565    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-oauth-serving-cert\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.748687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748662    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.748911 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748682    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.748911 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748697    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-registration-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.748911 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748739    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-node-log\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.748911 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748887    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-run-ovn-kubernetes\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.749103 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748913    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-multus-certs\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.749103 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748931    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-system-cni-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.749103 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748957    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.749103 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.748969    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-grpc-tls\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.749103 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749027    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.749103 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749054    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-serving-cert\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.749103 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749073    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-socket-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.749103 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749090    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kck4b\" (UniqueName: \"kubernetes.io/projected/f683dbce-ebac-4d2c-ba04-aba3ac20792b-kube-api-access-kck4b\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.749490 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749116    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkc5w\" (UniqueName: \"kubernetes.io/projected/baaab207-5534-4ef9-ac3a-9466ea5b900e-kube-api-access-hkc5w\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.749490 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749168    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-textfile\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.749490 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749254    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-grpc-tls\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.749490 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749283    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-textfile\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.749490 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749301    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-config\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.749490 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749335    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-oauth-serving-cert\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.749490 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749344    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-serving-cert\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.749490 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749359    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.749490 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749485    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtbk\" (UniqueName: \"kubernetes.io/projected/3017c536-859c-4b7b-b59e-f1d5ee77e117-kube-api-access-zvtbk\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749524    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-trusted-ca-bundle\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749533    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-oauth-serving-cert\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749554    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749563    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-etc-selinux\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749593    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749649    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbe53da9-f387-477c-8591-f5c37f42eaf5-tmp\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749685    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749733    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749765    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-cni-binary-copy\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749789    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-systemd\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749818    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3accc688-8a04-4462-9593-228a18548478-config-volume\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749845    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-kubernetes\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749870    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749882    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749896    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-service-ca\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.749926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749920    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbcb\" (UniqueName: \"kubernetes.io/projected/5183b0be-c47a-4e35-a8cb-77991e6f7208-kube-api-access-zpbcb\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749946    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-systemd\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749970    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-cnibin\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.749995    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79bf4395-e3d6-4071-aedd-548c185fa39d-data-volume\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750063    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmrsr\" (UniqueName: \"kubernetes.io/projected/79bf4395-e3d6-4071-aedd-548c185fa39d-kube-api-access-kmrsr\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750077    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750098    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drx2d\" (UniqueName: \"kubernetes.io/projected/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-kube-api-access-drx2d\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750109    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-cni-binary-copy\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750103    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750218    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79bf4395-e3d6-4071-aedd-548c185fa39d-data-volume\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750219    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbe53da9-f387-477c-8591-f5c37f42eaf5-tmp\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750229    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-config\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750293    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3accc688-8a04-4462-9593-228a18548478-config-volume\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750317    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-service-ca\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750329    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750366    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750410    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750451    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-systemd-units\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750459    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750484    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-ovnkube-config\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750507    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef2a700a-5061-402a-ab9c-29a44204054c-metrics-certs\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750584    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-tuned\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750587    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750614    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-wtmp\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750668    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-tuned\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750673    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-oauth-config\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750704    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-tls\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.750727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750707    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-ovnkube-config\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750759    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750788    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-serving-cert\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750807    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-system-cni-dir\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750833    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-cni-netd\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750857    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysctl-d\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750885    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-oauth-config\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750938    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750975    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.750990    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-tls\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751024    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751089    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-serving-cert\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751093    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751107    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751140    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c2dd590-c639-42fe-8a02-4945b6403b3b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wc4l2\" (UID: \"4c2dd590-c639-42fe-8a02-4945b6403b3b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751143    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751168    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-tls\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751218    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht42j\" (UniqueName: \"kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j\") pod \"network-check-target-n8m99\" (UID: \"0fadb982-a597-4489-ae16-3dc4fb5a7810\") " pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751257    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751323    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-env-overrides\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751343    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c2dd590-c639-42fe-8a02-4945b6403b3b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wc4l2\" (UID: \"4c2dd590-c639-42fe-8a02-4945b6403b3b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751404    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-ovnkube-script-lib\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751421    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-tls\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751474    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3accc688-8a04-4462-9593-228a18548478-tmp-dir\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751523    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baaab207-5534-4ef9-ac3a-9466ea5b900e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751538    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-env-overrides\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751545    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3accc688-8a04-4462-9593-228a18548478-tmp-dir\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751580    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-federate-client-tls\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751653    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/801f804c-187a-4232-af07-412a115a2fef-ovnkube-script-lib\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751668    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79bf4395-e3d6-4071-aedd-548c185fa39d-crio-socket\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:47:08.751779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751694    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b96d72b2-bc45-4962-b529-f08155ad0215-metrics-client-ca\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751817    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-federate-client-tls\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751862    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b96d72b2-bc45-4962-b529-f08155ad0215-tmp\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751875    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b96d72b2-bc45-4962-b529-f08155ad0215-metrics-client-ca\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751864    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/baaab207-5534-4ef9-ac3a-9466ea5b900e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751915    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b96d72b2-bc45-4962-b529-f08155ad0215-tmp\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751888    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz6vz\" (UniqueName: \"kubernetes.io/projected/2f64c76d-6e73-4cc0-9043-8f6235e0e281-kube-api-access-sz6vz\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751953    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-cni-bin\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.751990    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtl8\" (UniqueName: \"kubernetes.io/projected/ef2a700a-5061-402a-ab9c-29a44204054c-kube-api-access-tbtl8\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752026    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752049    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-oauth-config\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752091    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-telemeter-client-tls\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752113    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5fvk\" (UniqueName: \"kubernetes.io/projected/4c2dd590-c639-42fe-8a02-4945b6403b3b-kube-api-access-t5fvk\") pod \"monitoring-plugin-77f4f76594-wc4l2\" (UID: \"4c2dd590-c639-42fe-8a02-4945b6403b3b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752130    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysconfig\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752146    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-cnibin\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752164    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-var-lib-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752186    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-log-socket\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752220    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-client-ca-bundle\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752265    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-oauth-config\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752271    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752319    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-modprobe-d\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752364    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79bf4395-e3d6-4071-aedd-548c185fa39d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752390    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-client-ca-bundle\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752395    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-web-config\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752401    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-telemeter-client-tls\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752438    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-kubelet\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752543    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09b130d-4249-4539-add0-e0aba69b752a-tmp-dir\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752577    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6b42\" (UniqueName: \"kubernetes.io/projected/cd3c4499-8959-4616-842b-c556e7d23137-kube-api-access-b6b42\") pod \"ingress-canary-zv5gd\" (UID: \"cd3c4499-8959-4616-842b-c556e7d23137\") " pod="openshift-ingress-canary/ingress-canary-zv5gd"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752605    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752646    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79bf4395-e3d6-4071-aedd-548c185fa39d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752650    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752686    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klh9r\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-kube-api-access-klh9r\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752683    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09b130d-4249-4539-add0-e0aba69b752a-tmp-dir\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752656    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-web-config\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752709    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-run-netns\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752737    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-secret-metrics-server-tls\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752777    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vshbg\" (UniqueName: \"kubernetes.io/projected/c2e1c568-867f-472f-9c39-47588d5f3fca-kube-api-access-vshbg\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752799    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-lib-modules\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752800    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752815    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066041b1-7eba-4371-9470-101a7bf8fcfc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752834    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752852    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:47:08.752845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752876    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752903    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-ovn\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752933    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-service-ca\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752961    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-secret-telemeter-client\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752988    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9d2j\" (UniqueName: \"kubernetes.io/projected/dab249e6-5406-4ca6-87d5-d6fe41de7754-kube-api-access-t9d2j\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753015    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/baaab207-5534-4ef9-ac3a-9466ea5b900e-metrics-server-audit-profiles\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753021    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753040    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066041b1-7eba-4371-9470-101a7bf8fcfc-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753042    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753161    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-service-ca\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753172    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-metrics-client-ca\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753183    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dab249e6-5406-4ca6-87d5-d6fe41de7754-secret-telemeter-client\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753206    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnclr\" (UniqueName: \"kubernetes.io/projected/3accc688-8a04-4462-9593-228a18548478-kube-api-access-gnclr\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753208    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753235    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-netns\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.752998    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-secret-metrics-server-tls\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753260    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysctl-conf\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753265    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae94d17c-b881-46e1-8063-e88c9dc53369-original-pull-secret\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753283    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-host\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753331    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/baaab207-5534-4ef9-ac3a-9466ea5b900e-metrics-server-audit-profiles\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753336    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/324d72a4-a641-4b09-99d6-d682b5e2c958-agent-certs\") pod \"konnectivity-agent-vwk66\" (UID: \"324d72a4-a641-4b09-99d6-d682b5e2c958\") " pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753355    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-metrics-client-ca\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753378    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3017c536-859c-4b7b-b59e-f1d5ee77e117-host-slash\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753409    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753437    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/baaab207-5534-4ef9-ac3a-9466ea5b900e-audit-log\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753466    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-tmp\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753535    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/324d72a4-a641-4b09-99d6-d682b5e2c958-agent-certs\") pod \"konnectivity-agent-vwk66\" (UID: \"324d72a4-a641-4b09-99d6-d682b5e2c958\") " pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753541    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753574    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-tmp\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753586    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/baaab207-5534-4ef9-ac3a-9466ea5b900e-audit-log\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753601    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753667    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f64c76d-6e73-4cc0-9043-8f6235e0e281-tmp\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753709    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753734    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753756    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrkn\" (UniqueName: \"kubernetes.io/projected/1f41e102-fab5-4206-9502-fbf09515069a-kube-api-access-8vrkn\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753772    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f64c76d-6e73-4cc0-9043-8f6235e0e281-tmp\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753776    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753799    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-device-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753815    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-config\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753860    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcnrm\" (UniqueName: \"kubernetes.io/projected/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-kube-api-access-vcnrm\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753892    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f41e102-fab5-4206-9502-fbf09515069a-host\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753922    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753944    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-config\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.753961    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/801f804c-187a-4232-af07-412a115a2fef-ovn-node-metrics-cert\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754007    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae94d17c-b881-46e1-8063-e88c9dc53369-dbus\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754044    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-kubelet\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754085    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-cni-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754132    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgf24\" (UniqueName: \"kubernetes.io/projected/a09b130d-4249-4539-add0-e0aba69b752a-kube-api-access-rgf24\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754162    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd3c4499-8959-4616-842b-c556e7d23137-cert\") pod \"ingress-canary-zv5gd\" (UID: \"cd3c4499-8959-4616-842b-c556e7d23137\") " pod="openshift-ingress-canary/ingress-canary-zv5gd"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754186    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/801f804c-187a-4232-af07-412a115a2fef-ovn-node-metrics-cert\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754225    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-serving-cert\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754257    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754285    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-run\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754310    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-sys\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754336    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-var-lib-kubelet\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754388    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79bf4395-e3d6-4071-aedd-548c185fa39d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754394    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd3c4499-8959-4616-842b-c556e7d23137-cert\") pod \"ingress-canary-zv5gd\" (UID: \"cd3c4499-8959-4616-842b-c556e7d23137\") " pod="openshift-ingress-canary/ingress-canary-zv5gd"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754418    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754455    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-volume\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754471    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754487    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-serving-cert\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754523    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-oauth-config\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754558    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79bf4395-e3d6-4071-aedd-548c185fa39d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754601    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b96d72b2-bc45-4962-b529-f08155ad0215-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754608    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-cni-bin\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754680    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-daemon-config\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754699    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-sys\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754706    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-volume\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754736    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754743    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-oauth-config\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754755    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-etc-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754799    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-secret-metrics-server-client-certs\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754821    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-daemon-config\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754837    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae94d17c-b881-46e1-8063-e88c9dc53369-kubelet-config\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754919    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b64908a-fffb-4ec2-8697-51ea5db37f3b-metrics-client-ca\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754968    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3017c536-859c-4b7b-b59e-f1d5ee77e117-iptables-alerter-script\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.754977    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/baaab207-5534-4ef9-ac3a-9466ea5b900e-secret-metrics-server-client-certs\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755076    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b64908a-fffb-4ec2-8697-51ea5db37f3b-metrics-client-ca\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755073    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755180    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-slash\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755248    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5183b0be-c47a-4e35-a8cb-77991e6f7208-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755256    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755281    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3017c536-859c-4b7b-b59e-f1d5ee77e117-iptables-alerter-script\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755284    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-serving-certs-ca-bundle\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755315    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2e1c568-867f-472f-9c39-47588d5f3fca-cni-binary-copy\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755405    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-root\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755443    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dab249e6-5406-4ca6-87d5-d6fe41de7754-serving-certs-ca-bundle\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755496    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/066041b1-7eba-4371-9470-101a7bf8fcfc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755514    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkhr5\" (UniqueName: \"kubernetes.io/projected/3b64908a-fffb-4ec2-8697-51ea5db37f3b-kube-api-access-bkhr5\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755533    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755549    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-oauth-serving-cert\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755557    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/066041b1-7eba-4371-9470-101a7bf8fcfc-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755564    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-config\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755565    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2e1c568-867f-472f-9c39-47588d5f3fca-cni-binary-copy\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755585    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-conf-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755606    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-etc-kubernetes\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755644    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a09b130d-4249-4539-add0-e0aba69b752a-hosts-file\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755684    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8bm\" (UniqueName: \"kubernetes.io/projected/b96d72b2-bc45-4962-b529-f08155ad0215-kube-api-access-vv8bm\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755717    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-out\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755743    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-service-ca\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755764    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-oauth-serving-cert\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755770    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755771    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f41e102-fab5-4206-9502-fbf09515069a-serviceca\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755826    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-out\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755828    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-socket-dir-parent\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755855    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-sys-fs\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755906    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3accc688-8a04-4462-9593-228a18548478-metrics-tls\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755923    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-service-ca\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755945    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.755974    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7l79\" (UniqueName: \"kubernetes.io/projected/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-api-access-b7l79\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.756091    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3accc688-8a04-4462-9593-228a18548478-metrics-tls\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.756199    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-accelerators-collector-config\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.756205    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-config\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.757092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.756382    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f41e102-fab5-4206-9502-fbf09515069a-serviceca\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:47:08.759766 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.759740    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-trusted-ca-bundle\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.759863 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.759800    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.760018 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.759999    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-trusted-ca-bundle\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.771891 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.771860    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rfd8\" (UniqueName: \"kubernetes.io/projected/801f804c-187a-4232-af07-412a115a2fef-kube-api-access-7rfd8\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.772448 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.772432    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtbk\" (UniqueName: \"kubernetes.io/projected/3017c536-859c-4b7b-b59e-f1d5ee77e117-kube-api-access-zvtbk\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:47:08.774888 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.774865    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6m9n\" (UniqueName: \"kubernetes.io/projected/bbe53da9-f387-477c-8591-f5c37f42eaf5-kube-api-access-v6m9n\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.778141 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.778123    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz6vz\" (UniqueName: \"kubernetes.io/projected/2f64c76d-6e73-4cc0-9043-8f6235e0e281-kube-api-access-sz6vz\") pod \"console-968f5bf44-dhgp8\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") " pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:08.779732 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.779667    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht42j\" (UniqueName: \"kubernetes.io/projected/0fadb982-a597-4489-ae16-3dc4fb5a7810-kube-api-access-ht42j\") pod \"network-check-target-n8m99\" (UID: \"0fadb982-a597-4489-ae16-3dc4fb5a7810\") " pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:47:08.779879 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.779771    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwrg\" (UniqueName: \"kubernetes.io/projected/8c893b43-cd05-45a0-9f7b-a2b5d72809c7-kube-api-access-ldwrg\") pod \"openshift-state-metrics-7bd7489957-js76b\" (UID: \"8c893b43-cd05-45a0-9f7b-a2b5d72809c7\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-js76b"
Oct 29 21:47:08.779999 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.779979    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbcb\" (UniqueName: \"kubernetes.io/projected/5183b0be-c47a-4e35-a8cb-77991e6f7208-kube-api-access-zpbcb\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.780089 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.780054    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kck4b\" (UniqueName: \"kubernetes.io/projected/f683dbce-ebac-4d2c-ba04-aba3ac20792b-kube-api-access-kck4b\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.780147 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.780087    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmrsr\" (UniqueName: \"kubernetes.io/projected/79bf4395-e3d6-4071-aedd-548c185fa39d-kube-api-access-kmrsr\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:47:08.780147 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.780105    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrkn\" (UniqueName: \"kubernetes.io/projected/1f41e102-fab5-4206-9502-fbf09515069a-kube-api-access-8vrkn\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:47:08.780245 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.780231    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkc5w\" (UniqueName: \"kubernetes.io/projected/baaab207-5534-4ef9-ac3a-9466ea5b900e-kube-api-access-hkc5w\") pod \"metrics-server-f7fc46b4-vvxlv\" (UID: \"baaab207-5534-4ef9-ac3a-9466ea5b900e\") " pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:08.780288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.780239    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drx2d\" (UniqueName: \"kubernetes.io/projected/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-kube-api-access-drx2d\") pod \"console-779475d747-twkqw\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") " pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:08.781221 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.781195    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vshbg\" (UniqueName: \"kubernetes.io/projected/c2e1c568-867f-472f-9c39-47588d5f3fca-kube-api-access-vshbg\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.781387 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.781371    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgf24\" (UniqueName: \"kubernetes.io/projected/a09b130d-4249-4539-add0-e0aba69b752a-kube-api-access-rgf24\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:47:08.784223 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.784200    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkhr5\" (UniqueName: \"kubernetes.io/projected/3b64908a-fffb-4ec2-8697-51ea5db37f3b-kube-api-access-bkhr5\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.784482 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.784447    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9d2j\" (UniqueName: \"kubernetes.io/projected/dab249e6-5406-4ca6-87d5-d6fe41de7754-kube-api-access-t9d2j\") pod \"telemeter-client-54969c98b4-7zdlx\" (UID: \"dab249e6-5406-4ca6-87d5-d6fe41de7754\") " pod="openshift-monitoring/telemeter-client-54969c98b4-7zdlx"
Oct 29 21:47:08.789078 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.789048    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5fvk\" (UniqueName: \"kubernetes.io/projected/4c2dd590-c639-42fe-8a02-4945b6403b3b-kube-api-access-t5fvk\") pod \"monitoring-plugin-77f4f76594-wc4l2\" (UID: \"4c2dd590-c639-42fe-8a02-4945b6403b3b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:47:08.789223 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.789048    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6b42\" (UniqueName: \"kubernetes.io/projected/cd3c4499-8959-4616-842b-c556e7d23137-kube-api-access-b6b42\") pod \"ingress-canary-zv5gd\" (UID: \"cd3c4499-8959-4616-842b-c556e7d23137\") " pod="openshift-ingress-canary/ingress-canary-zv5gd"
Oct 29 21:47:08.789223 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.789129    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcnrm\" (UniqueName: \"kubernetes.io/projected/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-kube-api-access-vcnrm\") pod \"console-6c7d74bc46-5bb2n\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") " pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:08.789350 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.789229    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnclr\" (UniqueName: \"kubernetes.io/projected/3accc688-8a04-4462-9593-228a18548478-kube-api-access-gnclr\") pod \"dns-default-245p6\" (UID: \"3accc688-8a04-4462-9593-228a18548478\") " pod="openshift-dns/dns-default-245p6"
Oct 29 21:47:08.789671 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.789621    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbtl8\" (UniqueName: \"kubernetes.io/projected/ef2a700a-5061-402a-ab9c-29a44204054c-kube-api-access-tbtl8\") pod \"network-metrics-daemon-4zxrn\" (UID: \"ef2a700a-5061-402a-ab9c-29a44204054c\") " pod="openshift-multus/network-metrics-daemon-4zxrn"
Oct 29 21:47:08.789827 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.789808    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8bm\" (UniqueName: \"kubernetes.io/projected/b96d72b2-bc45-4962-b529-f08155ad0215-kube-api-access-vv8bm\") pod \"thanos-querier-f96f5cd85-t7ndb\" (UID: \"b96d72b2-bc45-4962-b529-f08155ad0215\") " pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:08.789890 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.789876    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7l79\" (UniqueName: \"kubernetes.io/projected/066041b1-7eba-4371-9470-101a7bf8fcfc-kube-api-access-b7l79\") pod \"kube-state-metrics-5dc8d7d89b-xw2d4\" (UID: \"066041b1-7eba-4371-9470-101a7bf8fcfc\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xw2d4"
Oct 29 21:47:08.789954 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.789939    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klh9r\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-kube-api-access-klh9r\") pod \"alertmanager-main-0\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:08.821757 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.821718    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-779475d747-twkqw"]
Oct 29 21:47:08.857324 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857230    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-systemd\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.857324 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857269    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-cnibin\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.857324 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857292    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-systemd-units\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.857324 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857312    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-wtmp\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857336    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-system-cni-dir\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857357    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-cni-netd\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857374    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysctl-d\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857382    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-systemd\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857382    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-cnibin\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857419    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-system-cni-dir\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857418    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-cni-netd\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857381    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-systemd-units\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857442    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-node-exporter-wtmp\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857449    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79bf4395-e3d6-4071-aedd-548c185fa39d-crio-socket\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857487    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79bf4395-e3d6-4071-aedd-548c185fa39d-crio-socket\") pod \"insights-runtime-extractor-2867l\" (UID: \"79bf4395-e3d6-4071-aedd-548c185fa39d\") " pod="openshift-insights/insights-runtime-extractor-2867l"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857520    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-cni-bin\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857538    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysctl-d\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857544    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-cni-bin\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857550    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysconfig\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857591    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-cnibin\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.857640 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857620    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-var-lib-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857660    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-log-socket\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857683    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-cnibin\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857690    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-var-lib-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857684    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-modprobe-d\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857596    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysconfig\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857710    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-log-socket\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857727    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-kubelet\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857755    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-run-netns\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857785    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-lib-modules\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857794    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-kubelet\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857784    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-modprobe-d\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857808    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-run-netns\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857825    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-ovn\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857840    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-ovn\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857860    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-netns\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857885    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysctl-conf\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857917    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-lib-modules\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857924    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-host\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857954    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3017c536-859c-4b7b-b59e-f1d5ee77e117-host-slash\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857967    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-host\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857953    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-netns\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857982    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-device-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858021    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-device-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858031    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f41e102-fab5-4206-9502-fbf09515069a-host\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.857984    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3017c536-859c-4b7b-b59e-f1d5ee77e117-host-slash\") pod \"iptables-alerter-l7z5g\" (UID: \"3017c536-859c-4b7b-b59e-f1d5ee77e117\") " pod="openshift-network-operator/iptables-alerter-l7z5g"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858054    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-sysctl-conf\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858059    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858079    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f41e102-fab5-4206-9502-fbf09515069a-host\") pod \"node-ca-dmgg8\" (UID: \"1f41e102-fab5-4206-9502-fbf09515069a\") " pod="openshift-image-registry/node-ca-dmgg8"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858090    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858100    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae94d17c-b881-46e1-8063-e88c9dc53369-dbus\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858118    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-kubelet\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858147    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-cni-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858172    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-run\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858180    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-kubelet\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858193    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-sys\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858218    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-var-lib-kubelet\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858240    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-cni-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858226    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae94d17c-b881-46e1-8063-e88c9dc53369-dbus\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858268    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-sys\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858250    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-cni-bin\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.858269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858281    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-cni-bin\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858298    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-var-lib-kubelet\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858312    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-sys\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858339    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-sys\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858366    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858393    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-etc-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858232    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-run\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858426    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae94d17c-b881-46e1-8063-e88c9dc53369-kubelet-config\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858453    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-etc-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858465    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-slash\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858480    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858493    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858500    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-slash\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858482    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae94d17c-b881-46e1-8063-e88c9dc53369-kubelet-config\") pod \"global-pull-secret-syncer-vth7d\" (UID: \"ae94d17c-b881-46e1-8063-e88c9dc53369\") " pod="kube-system/global-pull-secret-syncer-vth7d"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858517    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-root\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858583    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-conf-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858602    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-etc-kubernetes\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858618    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a09b130d-4249-4539-add0-e0aba69b752a-hosts-file\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858666    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-socket-dir-parent\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858675    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-conf-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858689    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-sys-fs\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858697    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-etc-kubernetes\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858714    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-hostroot\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858723    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a09b130d-4249-4539-add0-e0aba69b752a-hosts-file\") pod \"node-resolver-8jkmb\" (UID: \"a09b130d-4249-4539-add0-e0aba69b752a\") " pod="openshift-dns/node-resolver-8jkmb"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858730    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-multus-socket-dir-parent\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858743    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858751    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-sys-fs\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858777    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-k8s-cni-cncf-io\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858789    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-hostroot\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858801    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-cni-multus\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858824    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-k8s-cni-cncf-io\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858828    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-registration-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858778    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5183b0be-c47a-4e35-a8cb-77991e6f7208-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd2nd\" (UID: \"5183b0be-c47a-4e35-a8cb-77991e6f7208\") " pod="openshift-multus/multus-additional-cni-plugins-gd2nd"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858835    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-var-lib-cni-multus\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858860    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-node-log\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858883    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-registration-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858557    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b64908a-fffb-4ec2-8697-51ea5db37f3b-root\") pod \"node-exporter-wtbq7\" (UID: \"3b64908a-fffb-4ec2-8697-51ea5db37f3b\") " pod="openshift-monitoring/node-exporter-wtbq7"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858929    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-run-ovn-kubernetes\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858972    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-multus-certs\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.859080    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-host-run-ovn-kubernetes\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.859117    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-system-cni-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.859148    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-socket-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.859182    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-etc-selinux\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.859212    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-systemd\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.859236    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-kubernetes\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858522    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-openvswitch\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.859327    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbe53da9-f387-477c-8591-f5c37f42eaf5-etc-kubernetes\") pod \"tuned-frxl2\" (UID: \"bbe53da9-f387-477c-8591-f5c37f42eaf5\") " pod="openshift-cluster-node-tuning-operator/tuned-frxl2"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.859380    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-socket-dir\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.859379    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f683dbce-ebac-4d2c-ba04-aba3ac20792b-etc-selinux\") pod \"aws-ebs-csi-driver-node-5787g\" (UID: \"f683dbce-ebac-4d2c-ba04-aba3ac20792b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.859412    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-run-systemd\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.859427    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-host-run-multus-certs\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.859910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.859449    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2e1c568-867f-472f-9c39-47588d5f3fca-system-cni-dir\") pod \"multus-n4nsq\" (UID: \"c2e1c568-867f-472f-9c39-47588d5f3fca\") " pod="openshift-multus/multus-n4nsq"
Oct 29 21:47:08.861307 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:08.858916    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/801f804c-187a-4232-af07-412a115a2fef-node-log\") pod \"ovnkube-node-pzj9l\" (UID: \"801f804c-187a-4232-af07-412a115a2fef\") " pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:10.027756 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:10.027714    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-245p6"
Oct 29 21:47:10.926257 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:10.926228    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb"
Oct 29 21:47:12.612575 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:12.612539    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:13.934650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:13.934597    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:14.157647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:14.157612    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:14.695598 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:14.695560    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:14.699492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:14.699464    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:14.737438 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:14.737407    7082 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:47:16.519355 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:16.519294    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:47:16.849310 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:16.849276    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2"
Oct 29 21:47:18.425057 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:18.425023    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:21.184112 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:21.184072    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vwk66"
Oct 29 21:47:28.413168 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:28.413128    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:47:28.430129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:28.430100    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:47:29.774908 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:29.774874    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv"
Oct 29 21:47:33.661829 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.661778    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c7d74bc46-5bb2n" podUID="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" containerName="console" containerID="cri-o://3e24598f21593450c2d74dd245c7e519161ac49e41a357f860e17fc275e3dead" gracePeriod=15
Oct 29 21:47:33.813651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.813604    7082 generic.go:356] "Generic (PLEG): container finished" podID="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" containerID="3e24598f21593450c2d74dd245c7e519161ac49e41a357f860e17fc275e3dead" exitCode=0
Oct 29 21:47:33.813825 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.813683    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7d74bc46-5bb2n" event={"ID":"c20e4d88-bb47-4294-8242-2e71ca6d3f6c","Type":"ContainerDied","Data":"3e24598f21593450c2d74dd245c7e519161ac49e41a357f860e17fc275e3dead"}
Oct 29 21:47:33.919208 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.919126    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:33.973895 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.973862    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-config\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-config\") pod \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") "
Oct 29 21:47:33.974102 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.973927    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-kube-api-access-vcnrm\" (UniqueName: \"kubernetes.io/projected/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-kube-api-access-vcnrm\") pod \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") "
Oct 29 21:47:33.974102 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.974009    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-service-ca\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-service-ca\") pod \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") "
Oct 29 21:47:33.974102 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.974041    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-oauth-config\") pod \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") "
Oct 29 21:47:33.974102 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.974071    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-tmp\") pod \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") "
Oct 29 21:47:33.974305 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.974124    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-oauth-serving-cert\") pod \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") "
Oct 29 21:47:33.974305 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.974165    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-trusted-ca-bundle\") pod \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") "
Oct 29 21:47:33.974305 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.974202    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-serving-cert\") pod \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\" (UID: \"c20e4d88-bb47-4294-8242-2e71ca6d3f6c\") "
Oct 29 21:47:33.974446 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.974309    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-config" pod "c20e4d88-bb47-4294-8242-2e71ca6d3f6c" (UID: "c20e4d88-bb47-4294-8242-2e71ca6d3f6c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:33.974553 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.974515    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-oauth-serving-cert" pod "c20e4d88-bb47-4294-8242-2e71ca6d3f6c" (UID: "c20e4d88-bb47-4294-8242-2e71ca6d3f6c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:33.974600 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.974532    7082 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-config\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:33.974655 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.974584    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-trusted-ca-bundle" pod "c20e4d88-bb47-4294-8242-2e71ca6d3f6c" (UID: "c20e4d88-bb47-4294-8242-2e71ca6d3f6c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:33.975142 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.974956    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-service-ca" pod "c20e4d88-bb47-4294-8242-2e71ca6d3f6c" (UID: "c20e4d88-bb47-4294-8242-2e71ca6d3f6c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:33.975142 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.974963    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-tmp" pod "c20e4d88-bb47-4294-8242-2e71ca6d3f6c" (UID: "c20e4d88-bb47-4294-8242-2e71ca6d3f6c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:47:33.976847 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.976818    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-serving-cert" pod "c20e4d88-bb47-4294-8242-2e71ca6d3f6c" (UID: "c20e4d88-bb47-4294-8242-2e71ca6d3f6c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:33.977176 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.977148    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-kube-api-access-vcnrm" pod "c20e4d88-bb47-4294-8242-2e71ca6d3f6c" (UID: "c20e4d88-bb47-4294-8242-2e71ca6d3f6c"). InnerVolumeSpecName "kube-api-access-vcnrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:47:33.977275 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:33.977248    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-oauth-config" pod "c20e4d88-bb47-4294-8242-2e71ca6d3f6c" (UID: "c20e4d88-bb47-4294-8242-2e71ca6d3f6c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:34.075045 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.075010    7082 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:34.075045 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.075050    7082 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-oauth-config\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:34.075274 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.075062    7082 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-tmp\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:34.075274 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.075072    7082 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-oauth-serving-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:34.075274 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.075081    7082 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-trusted-ca-bundle\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:34.075274 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.075089    7082 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-console-serving-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:34.075274 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.075099    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcnrm\" (UniqueName: \"kubernetes.io/projected/c20e4d88-bb47-4294-8242-2e71ca6d3f6c-kube-api-access-vcnrm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:34.735239 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.735178    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-779475d747-twkqw" podUID="a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" containerName="console" containerID="cri-o://7bd4054d7b4256f685efd5ef8fedbb8141a441169fcb2bdaaa771d8911af1108" gracePeriod=15
Oct 29 21:47:34.819111 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.819077    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7d74bc46-5bb2n" event={"ID":"c20e4d88-bb47-4294-8242-2e71ca6d3f6c","Type":"ContainerDied","Data":"dbf589c6e0dabaec0d066764594086a44f6bc30d6f4d74670ad6dca76ed2ad62"}
Oct 29 21:47:34.819267 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.819139    7082 scope.go:122] "RemoveContainer" containerID="3e24598f21593450c2d74dd245c7e519161ac49e41a357f860e17fc275e3dead"
Oct 29 21:47:34.819267 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.819168    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7d74bc46-5bb2n"
Oct 29 21:47:34.869256 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.869224    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7d74bc46-5bb2n"]
Oct 29 21:47:34.881434 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.881324    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c7d74bc46-5bb2n"]
Oct 29 21:47:34.987404 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:34.987347    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:35.083920 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.083886    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-oauth-config\") pod \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") "
Oct 29 21:47:35.084124 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.083943    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-tmp\") pod \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") "
Oct 29 21:47:35.084124 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.083983    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-serving-cert\") pod \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") "
Oct 29 21:47:35.084124 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.084017    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-kube-api-access-drx2d\" (UniqueName: \"kubernetes.io/projected/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-kube-api-access-drx2d\") pod \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") "
Oct 29 21:47:35.084124 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.084050    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-oauth-serving-cert\") pod \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") "
Oct 29 21:47:35.084124 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.084087    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-config\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-config\") pod \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") "
Oct 29 21:47:35.084383 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.084127    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-service-ca\") pod \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\" (UID: \"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f\") "
Oct 29 21:47:35.084383 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.084285    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-tmp" pod "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" (UID: "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:47:35.084460 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.084445    7082 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-tmp\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:35.084585 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.084569    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-oauth-serving-cert" pod "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" (UID: "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:35.084620 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.084595    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-config" pod "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" (UID: "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:35.084695 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.084671    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-service-ca" pod "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" (UID: "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:35.086400 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.086378    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-oauth-config" pod "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" (UID: "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:35.086502 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.086427    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-kube-api-access-drx2d" pod "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" (UID: "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f"). InnerVolumeSpecName "kube-api-access-drx2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:47:35.086560 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.086526    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-serving-cert" pod "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" (UID: "a3f0e68b-09b9-49e8-bce4-cae4e15ec43f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:35.184983 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.184947    7082 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-serving-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:35.184983 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.184978    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drx2d\" (UniqueName: \"kubernetes.io/projected/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-kube-api-access-drx2d\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:35.184983 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.184989    7082 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-oauth-serving-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:35.185461 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.184998    7082 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-config\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:35.185461 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.185008    7082 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:35.185461 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.185016    7082 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f-console-oauth-config\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:35.523540 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.523510    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pzj9l"
Oct 29 21:47:35.628523 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.628481    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" path="/var/lib/kubelet/pods/c20e4d88-bb47-4294-8242-2e71ca6d3f6c/volumes"
Oct 29 21:47:35.825087 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.825055    7082 generic.go:356] "Generic (PLEG): container finished" podID="a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" containerID="7bd4054d7b4256f685efd5ef8fedbb8141a441169fcb2bdaaa771d8911af1108" exitCode=0
Oct 29 21:47:35.825515 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.825105    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-779475d747-twkqw"
Oct 29 21:47:35.825515 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.825122    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-779475d747-twkqw" event={"ID":"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f","Type":"ContainerDied","Data":"7bd4054d7b4256f685efd5ef8fedbb8141a441169fcb2bdaaa771d8911af1108"}
Oct 29 21:47:35.825515 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.825167    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-779475d747-twkqw" event={"ID":"a3f0e68b-09b9-49e8-bce4-cae4e15ec43f","Type":"ContainerDied","Data":"2b7ac84a5de67fab87859f8db39452d0ad41c49da9b34418888d7ff01e658423"}
Oct 29 21:47:35.825515 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.825190    7082 scope.go:122] "RemoveContainer" containerID="7bd4054d7b4256f685efd5ef8fedbb8141a441169fcb2bdaaa771d8911af1108"
Oct 29 21:47:35.833295 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.833277    7082 scope.go:122] "RemoveContainer" containerID="7bd4054d7b4256f685efd5ef8fedbb8141a441169fcb2bdaaa771d8911af1108"
Oct 29 21:47:35.833574 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:35.833554    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd4054d7b4256f685efd5ef8fedbb8141a441169fcb2bdaaa771d8911af1108\": container with ID starting with 7bd4054d7b4256f685efd5ef8fedbb8141a441169fcb2bdaaa771d8911af1108 not found: ID does not exist" containerID="7bd4054d7b4256f685efd5ef8fedbb8141a441169fcb2bdaaa771d8911af1108"
Oct 29 21:47:35.833660 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.833587    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd4054d7b4256f685efd5ef8fedbb8141a441169fcb2bdaaa771d8911af1108"} err="failed to get container status \"7bd4054d7b4256f685efd5ef8fedbb8141a441169fcb2bdaaa771d8911af1108\": rpc error: code = NotFound desc = could not find container \"7bd4054d7b4256f685efd5ef8fedbb8141a441169fcb2bdaaa771d8911af1108\": container with ID starting with 7bd4054d7b4256f685efd5ef8fedbb8141a441169fcb2bdaaa771d8911af1108 not found: ID does not exist"
Oct 29 21:47:35.874300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.874267    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-779475d747-twkqw"]
Oct 29 21:47:35.885980 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:35.885950    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-779475d747-twkqw"]
Oct 29 21:47:36.521950 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:36.521923    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n8m99"
Oct 29 21:47:37.628432 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:37.628391    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" path="/var/lib/kubelet/pods/a3f0e68b-09b9-49e8-bce4-cae4e15ec43f/volumes"
Oct 29 21:47:46.856665 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:46.856612    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 21:47:46.857093 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:46.856967    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="alertmanager" containerID="cri-o://20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b" gracePeriod=120
Oct 29 21:47:46.857164 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:46.857063    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="kube-rbac-proxy-metric" containerID="cri-o://954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99" gracePeriod=120
Oct 29 21:47:46.857164 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:46.857082    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="kube-rbac-proxy" containerID="cri-o://c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2" gracePeriod=120
Oct 29 21:47:46.857276 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:46.857070    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="kube-rbac-proxy-web" containerID="cri-o://4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d" gracePeriod=120
Oct 29 21:47:46.857276 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:46.857097    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="config-reloader" containerID="cri-o://94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28" gracePeriod=120
Oct 29 21:47:46.857276 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:46.857244    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="prom-label-proxy" containerID="cri-o://9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e" gracePeriod=120
Oct 29 21:47:47.879353 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:47.879320    7082 generic.go:356] "Generic (PLEG): container finished" podID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerID="9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e" exitCode=0
Oct 29 21:47:47.879353 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:47.879347    7082 generic.go:356] "Generic (PLEG): container finished" podID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerID="c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2" exitCode=0
Oct 29 21:47:47.879353 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:47.879357    7082 generic.go:356] "Generic (PLEG): container finished" podID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerID="94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28" exitCode=0
Oct 29 21:47:47.879353 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:47.879363    7082 generic.go:356] "Generic (PLEG): container finished" podID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerID="20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b" exitCode=0
Oct 29 21:47:47.879859 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:47.879389    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerDied","Data":"9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e"}
Oct 29 21:47:47.879859 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:47.879459    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerDied","Data":"c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2"}
Oct 29 21:47:47.879859 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:47.879486    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerDied","Data":"94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28"}
Oct 29 21:47:47.879859 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:47.879511    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerDied","Data":"20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b"}
Oct 29 21:47:48.108713 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.108686    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:48.190592 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.190499    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-volume\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-volume\") pod \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") "
Oct 29 21:47:48.190592 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.190540    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-trusted-ca-bundle\") pod \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") "
Oct 29 21:47:48.190592 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.190568    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-cluster-tls-config\") pod \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") "
Oct 29 21:47:48.190592 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.190588    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-main-tls\") pod \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") "
Oct 29 21:47:48.190961 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.190612    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-web\") pod \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") "
Oct 29 21:47:48.190961 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.190668    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-out\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-out\") pod \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") "
Oct 29 21:47:48.190961 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.190688    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-web-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-web-config\") pod \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") "
Oct 29 21:47:48.190961 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.190734    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-metrics-client-ca\") pod \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") "
Oct 29 21:47:48.190961 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.190768    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-main-db\") pod \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") "
Oct 29 21:47:48.190961 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.190798    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-tls-assets\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-tls-assets\") pod \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") "
Oct 29 21:47:48.190961 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.190824    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy\") pod \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") "
Oct 29 21:47:48.190961 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.190855    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-kube-api-access-klh9r\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-kube-api-access-klh9r\") pod \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") "
Oct 29 21:47:48.190961 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.190897    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-metric\") pod \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\" (UID: \"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57\") "
Oct 29 21:47:48.191563 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.191078    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-trusted-ca-bundle" pod "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" (UID: "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:48.191563 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.191185    7082 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:48.191563 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.191349    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-metrics-client-ca" pod "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" (UID: "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:48.192888 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.192668    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-main-db" pod "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" (UID: "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:47:48.198901 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.198795    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-metric" pod "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" (UID: "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:48.202958 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.202907    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-tls-assets" pod "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" (UID: "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:47:48.203815 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.203748    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-web" pod "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" (UID: "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:48.203815 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.203761    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-out" pod "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" (UID: "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:47:48.204522 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.204483    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy" pod "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" (UID: "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:48.204953 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.204873    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-main-tls" pod "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" (UID: "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:48.206060 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.206020    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-kube-api-access-klh9r" pod "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" (UID: "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57"). InnerVolumeSpecName "kube-api-access-klh9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:47:48.206699 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.206661    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-volume" pod "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" (UID: "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:48.207838 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.207810    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-cluster-tls-config" pod "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" (UID: "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:48.217028 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.216986    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-web-config" pod "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" (UID: "31d414df-8bf2-4c06-b8f0-ef1b3ce29e57"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:48.292541 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.292494    7082 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-alertmanager-main-db\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:48.292541 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.292532    7082 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-tls-assets\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:48.292541 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.292545    7082 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:48.292808 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.292555    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klh9r\" (UniqueName: \"kubernetes.io/projected/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-kube-api-access-klh9r\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:48.292808 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.292565    7082 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:48.292808 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.292574    7082 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:48.292808 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.292582    7082 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-cluster-tls-config\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:48.292808 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.292591    7082 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-main-tls\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:48.292808 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.292600    7082 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:48.292808 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.292608    7082 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-config-out\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:48.292808 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.292664    7082 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-web-config\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:48.292808 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.292674    7082 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57-metrics-client-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:47:48.886605 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.886567    7082 generic.go:356] "Generic (PLEG): container finished" podID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerID="954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99" exitCode=0
Oct 29 21:47:48.886605 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.886594    7082 generic.go:356] "Generic (PLEG): container finished" podID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerID="4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d" exitCode=0
Oct 29 21:47:48.887078 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.886656    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerDied","Data":"954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99"}
Oct 29 21:47:48.887078 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.886687    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:48.887078 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.886733    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerDied","Data":"4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d"}
Oct 29 21:47:48.887078 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.886764    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31d414df-8bf2-4c06-b8f0-ef1b3ce29e57","Type":"ContainerDied","Data":"bed90a2833cf42ce4a410e6c930d6ef7667ac810586d5f076818e1b5ecaad60f"}
Oct 29 21:47:48.887078 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.886803    7082 scope.go:122] "RemoveContainer" containerID="9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e"
Oct 29 21:47:48.895481 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.895460    7082 scope.go:122] "RemoveContainer" containerID="954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99"
Oct 29 21:47:48.903650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.903602    7082 scope.go:122] "RemoveContainer" containerID="c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2"
Oct 29 21:47:48.911493 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.911472    7082 scope.go:122] "RemoveContainer" containerID="4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d"
Oct 29 21:47:48.919448 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.919426    7082 scope.go:122] "RemoveContainer" containerID="94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28"
Oct 29 21:47:48.927479 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.927458    7082 scope.go:122] "RemoveContainer" containerID="20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b"
Oct 29 21:47:48.941150 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.941129    7082 scope.go:122] "RemoveContainer" containerID="853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26"
Oct 29 21:47:48.949277 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.949255    7082 scope.go:122] "RemoveContainer" containerID="9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e"
Oct 29 21:47:48.949725 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:48.949611    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e\": container with ID starting with 9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e not found: ID does not exist" containerID="9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e"
Oct 29 21:47:48.949806 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.949739    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e"} err="failed to get container status \"9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e\": rpc error: code = NotFound desc = could not find container \"9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e\": container with ID starting with 9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e not found: ID does not exist"
Oct 29 21:47:48.949806 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.949764    7082 scope.go:122] "RemoveContainer" containerID="954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99"
Oct 29 21:47:48.950101 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:48.950074    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99\": container with ID starting with 954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99 not found: ID does not exist" containerID="954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99"
Oct 29 21:47:48.950151 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.950108    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99"} err="failed to get container status \"954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99\": rpc error: code = NotFound desc = could not find container \"954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99\": container with ID starting with 954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99 not found: ID does not exist"
Oct 29 21:47:48.950151 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.950124    7082 scope.go:122] "RemoveContainer" containerID="c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2"
Oct 29 21:47:48.950392 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:48.950370    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2\": container with ID starting with c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2 not found: ID does not exist" containerID="c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2"
Oct 29 21:47:48.950500 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.950397    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2"} err="failed to get container status \"c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2\": rpc error: code = NotFound desc = could not find container \"c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2\": container with ID starting with c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2 not found: ID does not exist"
Oct 29 21:47:48.950500 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.950417    7082 scope.go:122] "RemoveContainer" containerID="4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d"
Oct 29 21:47:48.950730 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:48.950697    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d\": container with ID starting with 4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d not found: ID does not exist" containerID="4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d"
Oct 29 21:47:48.950780 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.950739    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d"} err="failed to get container status \"4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d\": rpc error: code = NotFound desc = could not find container \"4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d\": container with ID starting with 4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d not found: ID does not exist"
Oct 29 21:47:48.950856 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.950782    7082 scope.go:122] "RemoveContainer" containerID="94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28"
Oct 29 21:47:48.951066 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:48.951025    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28\": container with ID starting with 94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28 not found: ID does not exist" containerID="94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28"
Oct 29 21:47:48.951113 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.951072    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28"} err="failed to get container status \"94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28\": rpc error: code = NotFound desc = could not find container \"94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28\": container with ID starting with 94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28 not found: ID does not exist"
Oct 29 21:47:48.951113 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.951087    7082 scope.go:122] "RemoveContainer" containerID="20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b"
Oct 29 21:47:48.951307 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:48.951287    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b\": container with ID starting with 20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b not found: ID does not exist" containerID="20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b"
Oct 29 21:47:48.951381 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.951315    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b"} err="failed to get container status \"20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b\": rpc error: code = NotFound desc = could not find container \"20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b\": container with ID starting with 20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b not found: ID does not exist"
Oct 29 21:47:48.951381 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.951335    7082 scope.go:122] "RemoveContainer" containerID="853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26"
Oct 29 21:47:48.951576 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:48.951552    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26\": container with ID starting with 853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26 not found: ID does not exist" containerID="853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26"
Oct 29 21:47:48.951576 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.951570    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26"} err="failed to get container status \"853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26\": rpc error: code = NotFound desc = could not find container \"853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26\": container with ID starting with 853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26 not found: ID does not exist"
Oct 29 21:47:48.951712 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.951586    7082 scope.go:122] "RemoveContainer" containerID="9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e"
Oct 29 21:47:48.951864 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.951838    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e"} err="failed to get container status \"9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e\": rpc error: code = NotFound desc = could not find container \"9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e\": container with ID starting with 9df1257a2f46dc21832d479fbbf439900286f2d8e6e93e91d5ed1faa4a35c40e not found: ID does not exist"
Oct 29 21:47:48.951864 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.951861    7082 scope.go:122] "RemoveContainer" containerID="954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99"
Oct 29 21:47:48.952145 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.952116    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99"} err="failed to get container status \"954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99\": rpc error: code = NotFound desc = could not find container \"954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99\": container with ID starting with 954ca567290db09b46bcc80db748dfc771c2660b69a25f816bd068cae4c63a99 not found: ID does not exist"
Oct 29 21:47:48.952145 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.952131    7082 scope.go:122] "RemoveContainer" containerID="c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2"
Oct 29 21:47:48.952376 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.952353    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2"} err="failed to get container status \"c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2\": rpc error: code = NotFound desc = could not find container \"c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2\": container with ID starting with c87d4a60f55a8ec8d3977434c668ddf7d214f0b460c2814c429f229d42591cf2 not found: ID does not exist"
Oct 29 21:47:48.952429 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.952378    7082 scope.go:122] "RemoveContainer" containerID="4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d"
Oct 29 21:47:48.952586 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.952565    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d"} err="failed to get container status \"4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d\": rpc error: code = NotFound desc = could not find container \"4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d\": container with ID starting with 4ceaa9450cf3e81f55cd24fdcf22d2a6bfc311b0933813278184325ccfbcbc2d not found: ID does not exist"
Oct 29 21:47:48.952719 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.952586    7082 scope.go:122] "RemoveContainer" containerID="94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28"
Oct 29 21:47:48.952813 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.952794    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28"} err="failed to get container status \"94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28\": rpc error: code = NotFound desc = could not find container \"94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28\": container with ID starting with 94c30d1a4ad2e35777ed58b36a33cc9ad3cd306f152686198d31f70b34a7bd28 not found: ID does not exist"
Oct 29 21:47:48.952867 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.952814    7082 scope.go:122] "RemoveContainer" containerID="20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b"
Oct 29 21:47:48.953028 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.953011    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b"} err="failed to get container status \"20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b\": rpc error: code = NotFound desc = could not find container \"20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b\": container with ID starting with 20f6cfaedb5fdbd49f3062f622e8d1eef2fc06f92c6cfdefc5b1e1e4936c317b not found: ID does not exist"
Oct 29 21:47:48.953028 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.953028    7082 scope.go:122] "RemoveContainer" containerID="853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26"
Oct 29 21:47:48.953116 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.953086    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 21:47:48.953272 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.953251    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26"} err="failed to get container status \"853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26\": rpc error: code = NotFound desc = could not find container \"853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26\": container with ID starting with 853fcc53e2de99c807418a8060f7166303ddd24b12efb510622e9335db28eb26 not found: ID does not exist"
Oct 29 21:47:48.964851 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.964819    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 21:47:48.995394 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995356    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 21:47:48.995681 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995668    7082 cpu_manager.go:414] "containerMap: removing container" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="alertmanager"
Oct 29 21:47:48.995721 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995683    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="alertmanager"
Oct 29 21:47:48.995721 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995696    7082 cpu_manager.go:414] "containerMap: removing container" podUID="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" containerName="console"
Oct 29 21:47:48.995721 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995702    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" containerName="console"
Oct 29 21:47:48.995721 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995710    7082 cpu_manager.go:414] "containerMap: removing container" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="prom-label-proxy"
Oct 29 21:47:48.995721 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995716    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="prom-label-proxy"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995723    7082 cpu_manager.go:414] "containerMap: removing container" podUID="a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" containerName="console"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995728    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" containerName="console"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995736    7082 cpu_manager.go:414] "containerMap: removing container" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="kube-rbac-proxy"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995741    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="kube-rbac-proxy"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995748    7082 cpu_manager.go:414] "containerMap: removing container" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="kube-rbac-proxy-web"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995752    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="kube-rbac-proxy-web"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995758    7082 cpu_manager.go:414] "containerMap: removing container" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="kube-rbac-proxy-metric"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995763    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="kube-rbac-proxy-metric"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995772    7082 cpu_manager.go:414] "containerMap: removing container" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="config-reloader"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995776    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="config-reloader"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995784    7082 cpu_manager.go:414] "containerMap: removing container" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="init-config-reloader"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995789    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="init-config-reloader"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995860    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="alertmanager"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995868    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="c20e4d88-bb47-4294-8242-2e71ca6d3f6c" containerName="console"
Oct 29 21:47:48.995869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995874    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="prom-label-proxy"
Oct 29 21:47:48.996279 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995880    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f0e68b-09b9-49e8-bce4-cae4e15ec43f" containerName="console"
Oct 29 21:47:48.996279 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995885    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="kube-rbac-proxy"
Oct 29 21:47:48.996279 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995890    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="kube-rbac-proxy-web"
Oct 29 21:47:48.996279 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995894    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="kube-rbac-proxy-metric"
Oct 29 21:47:48.996279 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995899    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="config-reloader"
Oct 29 21:47:48.996279 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.995904    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" containerName="init-config-reloader"
Oct 29 21:47:48.999321 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:48.999294    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.014327 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.014290    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 29 21:47:49.014327 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.014321    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 29 21:47:49.014604 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.014335    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 29 21:47:49.014604 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.014327    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 29 21:47:49.014604 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.014303    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 29 21:47:49.014604 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.014304    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 29 21:47:49.014604 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.014413    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 29 21:47:49.014604 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.014419    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9qchq\""
Oct 29 21:47:49.014604 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.014582    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 29 21:47:49.026551 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.026522    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 29 21:47:49.030735 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.030703    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 21:47:49.100142 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.100108    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7314f42f-afe2-4c74-8281-760de1e84ba6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.100332 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.100174    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7314f42f-afe2-4c74-8281-760de1e84ba6-config-out\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.100332 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.100193    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.100332 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.100267    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7314f42f-afe2-4c74-8281-760de1e84ba6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.100332 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.100324    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.100557 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.100349    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7314f42f-afe2-4c74-8281-760de1e84ba6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.100557 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.100391    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.100557 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.100440    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.100557 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.100456    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.100557 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.100480    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-config-volume\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.100557 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.100504    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-web-config\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.100557 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.100520    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvtgq\" (UniqueName: \"kubernetes.io/projected/7314f42f-afe2-4c74-8281-760de1e84ba6-kube-api-access-mvtgq\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.100863 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.100670    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7314f42f-afe2-4c74-8281-760de1e84ba6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.201253 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201167    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-config-volume\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.201253 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201207    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-web-config\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.201253 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201223    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvtgq\" (UniqueName: \"kubernetes.io/projected/7314f42f-afe2-4c74-8281-760de1e84ba6-kube-api-access-mvtgq\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.201253 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201241    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7314f42f-afe2-4c74-8281-760de1e84ba6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.201558 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201263    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7314f42f-afe2-4c74-8281-760de1e84ba6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.201558 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201314    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7314f42f-afe2-4c74-8281-760de1e84ba6-config-out\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.201558 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201338    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.201558 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201367    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7314f42f-afe2-4c74-8281-760de1e84ba6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.201558 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201391    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.201558 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201417    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7314f42f-afe2-4c74-8281-760de1e84ba6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.201558 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201440    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.201558 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201475    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.201558 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201497    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.202010 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201675    7082 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 21:47:49.202010 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.201984    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7314f42f-afe2-4c74-8281-760de1e84ba6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.202287 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.202242    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7314f42f-afe2-4c74-8281-760de1e84ba6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.203359 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.203329    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7314f42f-afe2-4c74-8281-760de1e84ba6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.204477 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.204447    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7314f42f-afe2-4c74-8281-760de1e84ba6-config-out\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.204670 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.204617    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7314f42f-afe2-4c74-8281-760de1e84ba6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.205004 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.204979    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.205142 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.205118    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.205210 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.205195    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-config-volume\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.205352 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.205332    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.205406 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.205350    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.206167 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.206148    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.206526 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.206507    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7314f42f-afe2-4c74-8281-760de1e84ba6-web-config\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.225736 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.225699    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvtgq\" (UniqueName: \"kubernetes.io/projected/7314f42f-afe2-4c74-8281-760de1e84ba6-kube-api-access-mvtgq\") pod \"alertmanager-main-0\" (UID: \"7314f42f-afe2-4c74-8281-760de1e84ba6\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.315560 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.315513    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 29 21:47:49.628724 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.628692    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d414df-8bf2-4c06-b8f0-ef1b3ce29e57" path="/var/lib/kubelet/pods/31d414df-8bf2-4c06-b8f0-ef1b3ce29e57/volumes"
Oct 29 21:47:49.659636 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.659587    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 29 21:47:49.663951 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:47:49.663919    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7314f42f_afe2_4c74_8281_760de1e84ba6.slice/crio-6c0f6cc2fcab2d82b411b8571eb8345f3bffb8c2189572e879e6dcfe83ebbbdb WatchSource:0}: Error finding container 6c0f6cc2fcab2d82b411b8571eb8345f3bffb8c2189572e879e6dcfe83ebbbdb: Status 404 returned error can't find the container with id 6c0f6cc2fcab2d82b411b8571eb8345f3bffb8c2189572e879e6dcfe83ebbbdb
Oct 29 21:47:49.892601 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.892505    7082 generic.go:356] "Generic (PLEG): container finished" podID="7314f42f-afe2-4c74-8281-760de1e84ba6" containerID="4520d9fda7a2d96ac48db07bdd70758e44d680b9173025e336a30742643f0fe5" exitCode=0
Oct 29 21:47:49.892601 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.892584    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7314f42f-afe2-4c74-8281-760de1e84ba6","Type":"ContainerDied","Data":"4520d9fda7a2d96ac48db07bdd70758e44d680b9173025e336a30742643f0fe5"}
Oct 29 21:47:49.893088 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:49.892672    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7314f42f-afe2-4c74-8281-760de1e84ba6","Type":"ContainerStarted","Data":"6c0f6cc2fcab2d82b411b8571eb8345f3bffb8c2189572e879e6dcfe83ebbbdb"}
Oct 29 21:47:50.900966 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:50.900927    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7314f42f-afe2-4c74-8281-760de1e84ba6","Type":"ContainerStarted","Data":"eaee3de850e4a1ec8d7b92dcafa30d74f0cf99a0d86b0ce5437aef110510f865"}
Oct 29 21:47:50.901370 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:50.901024    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7314f42f-afe2-4c74-8281-760de1e84ba6","Type":"ContainerStarted","Data":"0b7ecfa46bf536ff2703968c166fa7efab89a351cd613a4f701a78381ef57b2b"}
Oct 29 21:47:50.901370 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:50.901053    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7314f42f-afe2-4c74-8281-760de1e84ba6","Type":"ContainerStarted","Data":"640699a79de4768ddd1aa8a06f5027ae0475a741062ae2683e76211c4c92418c"}
Oct 29 21:47:50.901370 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:50.901079    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7314f42f-afe2-4c74-8281-760de1e84ba6","Type":"ContainerStarted","Data":"1f6008a17704813e6424a6b6978a87ac31c6479f47e40ce575ef5f74f08f79ff"}
Oct 29 21:47:50.901370 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:50.901103    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7314f42f-afe2-4c74-8281-760de1e84ba6","Type":"ContainerStarted","Data":"4c7503393064e2d43f306971abb4adc6e4d9b070837f9c841df7ddbe91844f9f"}
Oct 29 21:47:50.901370 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:50.901127    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7314f42f-afe2-4c74-8281-760de1e84ba6","Type":"ContainerStarted","Data":"4c764d81c73d3772832af1af530c2c06fde568155db63eb3d79f6a1a9f7ebcff"}
Oct 29 21:47:50.967973 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:47:50.967920    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.953838756 podStartE2EDuration="2.967906919s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.068163ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:50.966979044 +0000 UTC m=+43.467021365" watchObservedRunningTime="2026-10-29 21:47:50.967906919 +0000 UTC m=+43.467949217"
Oct 29 21:47:58.412371 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:47:58.412331    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:48:27.316191 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.316158    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bd87bd5df-9rfzx"]
Oct 29 21:48:27.319801 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.319780    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.332583 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.332551    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a9db242-c903-49e3-92a6-fdf982c8f25e-console-serving-cert\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.332769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.332607    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a9db242-c903-49e3-92a6-fdf982c8f25e-tmp\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.332769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.332703    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a9db242-c903-49e3-92a6-fdf982c8f25e-oauth-serving-cert\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.332769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.332730    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a9db242-c903-49e3-92a6-fdf982c8f25e-console-oauth-config\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.332769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.332764    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a9db242-c903-49e3-92a6-fdf982c8f25e-console-config\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.332898 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.332833    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9db242-c903-49e3-92a6-fdf982c8f25e-service-ca\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.332898 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.332850    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a9db242-c903-49e3-92a6-fdf982c8f25e-trusted-ca-bundle\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.332898 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.332864    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-498h4\" (UniqueName: \"kubernetes.io/projected/4a9db242-c903-49e3-92a6-fdf982c8f25e-kube-api-access-498h4\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.342713 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.342678    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bd87bd5df-9rfzx"]
Oct 29 21:48:27.434242 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.434206    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9db242-c903-49e3-92a6-fdf982c8f25e-service-ca\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.434242 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.434243    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a9db242-c903-49e3-92a6-fdf982c8f25e-trusted-ca-bundle\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.434492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.434258    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-498h4\" (UniqueName: \"kubernetes.io/projected/4a9db242-c903-49e3-92a6-fdf982c8f25e-kube-api-access-498h4\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.434492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.434279    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a9db242-c903-49e3-92a6-fdf982c8f25e-console-serving-cert\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.434492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.434324    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a9db242-c903-49e3-92a6-fdf982c8f25e-tmp\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.434492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.434351    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a9db242-c903-49e3-92a6-fdf982c8f25e-oauth-serving-cert\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.434492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.434381    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a9db242-c903-49e3-92a6-fdf982c8f25e-console-oauth-config\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.434492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.434424    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a9db242-c903-49e3-92a6-fdf982c8f25e-console-config\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.434878 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.434849    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a9db242-c903-49e3-92a6-fdf982c8f25e-tmp\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.435190 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.435161    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9db242-c903-49e3-92a6-fdf982c8f25e-service-ca\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.435458 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.435222    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a9db242-c903-49e3-92a6-fdf982c8f25e-console-config\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.435458 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.435304    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a9db242-c903-49e3-92a6-fdf982c8f25e-trusted-ca-bundle\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.435683 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.435661    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a9db242-c903-49e3-92a6-fdf982c8f25e-oauth-serving-cert\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.437289 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.437266    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a9db242-c903-49e3-92a6-fdf982c8f25e-console-serving-cert\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.437392 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.437292    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a9db242-c903-49e3-92a6-fdf982c8f25e-console-oauth-config\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.458259 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.458224    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-498h4\" (UniqueName: \"kubernetes.io/projected/4a9db242-c903-49e3-92a6-fdf982c8f25e-kube-api-access-498h4\") pod \"console-5bd87bd5df-9rfzx\" (UID: \"4a9db242-c903-49e3-92a6-fdf982c8f25e\") " pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.631253 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.631168    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:27.940119 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:27.940012    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bd87bd5df-9rfzx"]
Oct 29 21:48:27.944035 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:48:27.943990    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9db242_c903_49e3_92a6_fdf982c8f25e.slice/crio-45cc206d27c6b61a16aed395621c1ed9c4d71280b05ec5ce2083f9ae192a1361 WatchSource:0}: Error finding container 45cc206d27c6b61a16aed395621c1ed9c4d71280b05ec5ce2083f9ae192a1361: Status 404 returned error can't find the container with id 45cc206d27c6b61a16aed395621c1ed9c4d71280b05ec5ce2083f9ae192a1361
Oct 29 21:48:28.041846 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:28.041803    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bd87bd5df-9rfzx" event={"ID":"4a9db242-c903-49e3-92a6-fdf982c8f25e","Type":"ContainerStarted","Data":"a947bcb688960433f2f4b54721727451a41e4bf886bbf2720735a9e5accc1280"}
Oct 29 21:48:28.042024 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:28.041909    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bd87bd5df-9rfzx" event={"ID":"4a9db242-c903-49e3-92a6-fdf982c8f25e","Type":"ContainerStarted","Data":"45cc206d27c6b61a16aed395621c1ed9c4d71280b05ec5ce2083f9ae192a1361"}
Oct 29 21:48:28.085936 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:28.085885    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5bd87bd5df-9rfzx" podStartSLOduration=1.0858699569999999 podStartE2EDuration="1.085869957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:48:28.085351957 +0000 UTC m=+80.585394257" watchObservedRunningTime="2026-10-29 21:48:28.085869957 +0000 UTC m=+80.585912304"
Oct 29 21:48:28.411829 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:48:28.411789    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:48:37.632398 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:37.632345    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:37.633036 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:37.632590    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:37.637689 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:37.637664    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:38.085220 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:38.085188    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bd87bd5df-9rfzx"
Oct 29 21:48:38.161181 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:48:38.161147    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-968f5bf44-dhgp8"]
Oct 29 21:48:58.412926 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:48:58.412877    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:49:03.182016 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.181954    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-968f5bf44-dhgp8" podUID="2f64c76d-6e73-4cc0-9043-8f6235e0e281" containerName="console" containerID="cri-o://30669bfe69d6bd3c22b1dc115c5f2eec23d794755b761619f187baace29b5951" gracePeriod=15
Oct 29 21:49:03.436565 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.436484    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:49:03.462643 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.462586    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-trusted-ca-bundle\") pod \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") "
Oct 29 21:49:03.462845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.462676    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f64c76d-6e73-4cc0-9043-8f6235e0e281-kube-api-access-sz6vz\" (UniqueName: \"kubernetes.io/projected/2f64c76d-6e73-4cc0-9043-8f6235e0e281-kube-api-access-sz6vz\") pod \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") "
Oct 29 21:49:03.462845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.462716    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-config\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-config\") pod \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") "
Oct 29 21:49:03.462845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.462756    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-serving-cert\") pod \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") "
Oct 29 21:49:03.462845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.462787    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-oauth-serving-cert\") pod \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") "
Oct 29 21:49:03.463156 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.462852    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-oauth-config\") pod \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") "
Oct 29 21:49:03.463156 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.462902    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f64c76d-6e73-4cc0-9043-8f6235e0e281-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f64c76d-6e73-4cc0-9043-8f6235e0e281-tmp\") pod \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") "
Oct 29 21:49:03.463156 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.462943    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-service-ca\") pod \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\" (UID: \"2f64c76d-6e73-4cc0-9043-8f6235e0e281\") "
Oct 29 21:49:03.463301 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.463142    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-trusted-ca-bundle" pod "2f64c76d-6e73-4cc0-9043-8f6235e0e281" (UID: "2f64c76d-6e73-4cc0-9043-8f6235e0e281"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:49:03.463301 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.463185    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-oauth-serving-cert" pod "2f64c76d-6e73-4cc0-9043-8f6235e0e281" (UID: "2f64c76d-6e73-4cc0-9043-8f6235e0e281"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:49:03.463301 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.463196    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-config" pod "2f64c76d-6e73-4cc0-9043-8f6235e0e281" (UID: "2f64c76d-6e73-4cc0-9043-8f6235e0e281"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:49:03.463463 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.463337    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f64c76d-6e73-4cc0-9043-8f6235e0e281-tmp" pod "2f64c76d-6e73-4cc0-9043-8f6235e0e281" (UID: "2f64c76d-6e73-4cc0-9043-8f6235e0e281"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:03.463521 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.463497    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-service-ca" pod "2f64c76d-6e73-4cc0-9043-8f6235e0e281" (UID: "2f64c76d-6e73-4cc0-9043-8f6235e0e281"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:49:03.465374 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.465337    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-oauth-config" pod "2f64c76d-6e73-4cc0-9043-8f6235e0e281" (UID: "2f64c76d-6e73-4cc0-9043-8f6235e0e281"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:49:03.465735 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.465705    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-serving-cert" pod "2f64c76d-6e73-4cc0-9043-8f6235e0e281" (UID: "2f64c76d-6e73-4cc0-9043-8f6235e0e281"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:49:03.465735 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.465711    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f64c76d-6e73-4cc0-9043-8f6235e0e281-kube-api-access-sz6vz" pod "2f64c76d-6e73-4cc0-9043-8f6235e0e281" (UID: "2f64c76d-6e73-4cc0-9043-8f6235e0e281"). InnerVolumeSpecName "kube-api-access-sz6vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:49:03.564268 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.564218    7082 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f64c76d-6e73-4cc0-9043-8f6235e0e281-tmp\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:03.564268 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.564256    7082 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:03.564268 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.564269    7082 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-trusted-ca-bundle\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:03.564268 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.564278    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz6vz\" (UniqueName: \"kubernetes.io/projected/2f64c76d-6e73-4cc0-9043-8f6235e0e281-kube-api-access-sz6vz\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:03.564535 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.564288    7082 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-config\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:03.564535 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.564297    7082 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-serving-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:03.564535 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.564306    7082 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f64c76d-6e73-4cc0-9043-8f6235e0e281-oauth-serving-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:03.564535 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:03.564314    7082 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f64c76d-6e73-4cc0-9043-8f6235e0e281-console-oauth-config\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:04.179929 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:04.179895    7082 generic.go:356] "Generic (PLEG): container finished" podID="2f64c76d-6e73-4cc0-9043-8f6235e0e281" containerID="30669bfe69d6bd3c22b1dc115c5f2eec23d794755b761619f187baace29b5951" exitCode=0
Oct 29 21:49:04.180124 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:04.179941    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-968f5bf44-dhgp8"
Oct 29 21:49:04.180124 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:04.179972    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-968f5bf44-dhgp8" event={"ID":"2f64c76d-6e73-4cc0-9043-8f6235e0e281","Type":"ContainerDied","Data":"30669bfe69d6bd3c22b1dc115c5f2eec23d794755b761619f187baace29b5951"}
Oct 29 21:49:04.180124 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:04.180041    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-968f5bf44-dhgp8" event={"ID":"2f64c76d-6e73-4cc0-9043-8f6235e0e281","Type":"ContainerDied","Data":"60374fa76ee4647cf89872d467fec40c6a20477a19eccc4a213e21518f86c57a"}
Oct 29 21:49:04.180124 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:04.180061    7082 scope.go:122] "RemoveContainer" containerID="30669bfe69d6bd3c22b1dc115c5f2eec23d794755b761619f187baace29b5951"
Oct 29 21:49:04.191654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:04.191616    7082 scope.go:122] "RemoveContainer" containerID="30669bfe69d6bd3c22b1dc115c5f2eec23d794755b761619f187baace29b5951"
Oct 29 21:49:04.192016 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:49:04.191977    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30669bfe69d6bd3c22b1dc115c5f2eec23d794755b761619f187baace29b5951\": container with ID starting with 30669bfe69d6bd3c22b1dc115c5f2eec23d794755b761619f187baace29b5951 not found: ID does not exist" containerID="30669bfe69d6bd3c22b1dc115c5f2eec23d794755b761619f187baace29b5951"
Oct 29 21:49:04.192061 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:04.192029    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30669bfe69d6bd3c22b1dc115c5f2eec23d794755b761619f187baace29b5951"} err="failed to get container status \"30669bfe69d6bd3c22b1dc115c5f2eec23d794755b761619f187baace29b5951\": rpc error: code = NotFound desc = could not find container \"30669bfe69d6bd3c22b1dc115c5f2eec23d794755b761619f187baace29b5951\": container with ID starting with 30669bfe69d6bd3c22b1dc115c5f2eec23d794755b761619f187baace29b5951 not found: ID does not exist"
Oct 29 21:49:04.235264 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:04.235230    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-968f5bf44-dhgp8"]
Oct 29 21:49:04.246955 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:04.246917    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-968f5bf44-dhgp8"]
Oct 29 21:49:05.628344 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:05.628305    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f64c76d-6e73-4cc0-9043-8f6235e0e281" path="/var/lib/kubelet/pods/2f64c76d-6e73-4cc0-9043-8f6235e0e281/volumes"
Oct 29 21:49:20.976059 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:20.976013    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"]
Oct 29 21:49:20.976753 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:20.976381    7082 cpu_manager.go:414] "containerMap: removing container" podUID="2f64c76d-6e73-4cc0-9043-8f6235e0e281" containerName="console"
Oct 29 21:49:20.976753 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:20.976393    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f64c76d-6e73-4cc0-9043-8f6235e0e281" containerName="console"
Oct 29 21:49:20.976753 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:20.976482    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f64c76d-6e73-4cc0-9043-8f6235e0e281" containerName="console"
Oct 29 21:49:20.979812 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:20.979790    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"
Oct 29 21:49:21.005982 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.005937    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 29 21:49:21.005982 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.005967    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 29 21:49:21.006247 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.006163    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-ddg98\""
Oct 29 21:49:21.006770 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.006745    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"]
Oct 29 21:49:21.108035 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.107997    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs\" (UID: \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"
Oct 29 21:49:21.108239 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.108043    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-695m4\" (UniqueName: \"kubernetes.io/projected/4a48c8b5-1dd2-4533-8d64-63843a762d2e-kube-api-access-695m4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs\" (UID: \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"
Oct 29 21:49:21.108239 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.108083    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs\" (UID: \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"
Oct 29 21:49:21.181474 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.181440    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"]
Oct 29 21:49:21.184934 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.184909    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"
Oct 29 21:49:21.204380 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.204348    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-jpxcg\""
Oct 29 21:49:21.208603 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.208553    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs\" (UID: \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"
Oct 29 21:49:21.208603 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.208608    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-695m4\" (UniqueName: \"kubernetes.io/projected/4a48c8b5-1dd2-4533-8d64-63843a762d2e-kube-api-access-695m4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs\" (UID: \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"
Oct 29 21:49:21.208896 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.208703    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs\" (UID: \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"
Oct 29 21:49:21.209050 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.209028    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs\" (UID: \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"
Oct 29 21:49:21.209132 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.209093    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs\" (UID: \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"
Oct 29 21:49:21.209487 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.209465    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"]
Oct 29 21:49:21.235840 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.235765    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-695m4\" (UniqueName: \"kubernetes.io/projected/4a48c8b5-1dd2-4533-8d64-63843a762d2e-kube-api-access-695m4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs\" (UID: \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"
Oct 29 21:49:21.296816 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.296778    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"
Oct 29 21:49:21.309321 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.309290    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s\" (UID: \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"
Oct 29 21:49:21.309495 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.309361    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntkrn\" (UniqueName: \"kubernetes.io/projected/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-kube-api-access-ntkrn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s\" (UID: \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"
Oct 29 21:49:21.309495 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.309398    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s\" (UID: \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"
Oct 29 21:49:21.410349 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.410307    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s\" (UID: \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"
Oct 29 21:49:21.410572 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.410415    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntkrn\" (UniqueName: \"kubernetes.io/projected/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-kube-api-access-ntkrn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s\" (UID: \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"
Oct 29 21:49:21.410572 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.410445    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s\" (UID: \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"
Oct 29 21:49:21.410788 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.410767    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s\" (UID: \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"
Oct 29 21:49:21.410847 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.410791    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s\" (UID: \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"
Oct 29 21:49:21.434690 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.434654    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntkrn\" (UniqueName: \"kubernetes.io/projected/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-kube-api-access-ntkrn\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s\" (UID: \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"
Oct 29 21:49:21.502419 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.502316    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"
Oct 29 21:49:21.604740 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.604695    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"]
Oct 29 21:49:21.608517 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:49:21.608486    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a48c8b5_1dd2_4533_8d64_63843a762d2e.slice/crio-f15e8d9167b3b2c1cea2b89b22272020ca346a51314cca2bea02255e43b45b2e WatchSource:0}: Error finding container f15e8d9167b3b2c1cea2b89b22272020ca346a51314cca2bea02255e43b45b2e: Status 404 returned error can't find the container with id f15e8d9167b3b2c1cea2b89b22272020ca346a51314cca2bea02255e43b45b2e
Oct 29 21:49:21.610518 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.610496    7082 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 21:49:21.814934 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:21.814897    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"]
Oct 29 21:49:21.818902 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:49:21.818868    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33547cfa_0e45_4620_b09c_bf6d98ec4f0a.slice/crio-78b0a038959090c55baac9b3f31a005b7695ab9800b1a6b340b8905890891178 WatchSource:0}: Error finding container 78b0a038959090c55baac9b3f31a005b7695ab9800b1a6b340b8905890891178: Status 404 returned error can't find the container with id 78b0a038959090c55baac9b3f31a005b7695ab9800b1a6b340b8905890891178
Oct 29 21:49:22.251030 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:22.250896    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs" event={"ID":"4a48c8b5-1dd2-4533-8d64-63843a762d2e","Type":"ContainerStarted","Data":"f15e8d9167b3b2c1cea2b89b22272020ca346a51314cca2bea02255e43b45b2e"}
Oct 29 21:49:22.252383 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:22.252330    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s" event={"ID":"33547cfa-0e45-4620-b09c-bf6d98ec4f0a","Type":"ContainerStarted","Data":"78b0a038959090c55baac9b3f31a005b7695ab9800b1a6b340b8905890891178"}
Oct 29 21:49:23.273891 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.271903    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"]
Oct 29 21:49:23.276150 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.276118    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"
Oct 29 21:49:23.303006 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.302935    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"fdb36248658848c45b2cd30265e4aa38f5747d-4cc5d9d1-dockercfg-sgbzm\""
Oct 29 21:49:23.303920 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.303829    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"]
Oct 29 21:49:23.336449 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.336407    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-util\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29\" (UID: \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"
Oct 29 21:49:23.336618 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.336482    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-bundle\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29\" (UID: \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"
Oct 29 21:49:23.336618 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.336504    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j6mp\" (UniqueName: \"kubernetes.io/projected/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-kube-api-access-5j6mp\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29\" (UID: \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"
Oct 29 21:49:23.437359 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.437318    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-bundle\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29\" (UID: \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"
Oct 29 21:49:23.437659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.437372    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j6mp\" (UniqueName: \"kubernetes.io/projected/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-kube-api-access-5j6mp\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29\" (UID: \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"
Oct 29 21:49:23.437659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.437426    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-util\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29\" (UID: \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"
Oct 29 21:49:23.438014 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.437933    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-util\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29\" (UID: \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"
Oct 29 21:49:23.438014 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.437961    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-bundle\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29\" (UID: \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"
Oct 29 21:49:23.462782 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.462739    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j6mp\" (UniqueName: \"kubernetes.io/projected/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-kube-api-access-5j6mp\") pod \"fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29\" (UID: \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\") " pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"
Oct 29 21:49:23.596984 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.596930    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"
Oct 29 21:49:23.922852 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:23.922762    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"]
Oct 29 21:49:24.265563 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:24.265420    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29" event={"ID":"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e","Type":"ContainerStarted","Data":"b4c32b12876ba5a91fca52e9cbcc470984e15f6da5c82b30baff8f72d16be842"}
Oct 29 21:49:24.899195 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:24.899156    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"]
Oct 29 21:49:24.903087 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:24.903057    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"
Oct 29 21:49:24.929684 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:24.929581    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-kg45n\""
Oct 29 21:49:24.930746 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:24.930717    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"]
Oct 29 21:49:24.955913 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:24.955579    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp\" (UID: \"d220ab29-136b-424f-ab41-af13009c041a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"
Oct 29 21:49:24.955913 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:24.955670    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp\" (UID: \"d220ab29-136b-424f-ab41-af13009c041a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"
Oct 29 21:49:24.955913 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:24.955741    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-922lk\" (UniqueName: \"kubernetes.io/projected/d220ab29-136b-424f-ab41-af13009c041a-kube-api-access-922lk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp\" (UID: \"d220ab29-136b-424f-ab41-af13009c041a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"
Oct 29 21:49:25.056280 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:25.056248    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-922lk\" (UniqueName: \"kubernetes.io/projected/d220ab29-136b-424f-ab41-af13009c041a-kube-api-access-922lk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp\" (UID: \"d220ab29-136b-424f-ab41-af13009c041a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"
Oct 29 21:49:25.056490 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:25.056328    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp\" (UID: \"d220ab29-136b-424f-ab41-af13009c041a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"
Oct 29 21:49:25.056490 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:25.056363    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp\" (UID: \"d220ab29-136b-424f-ab41-af13009c041a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"
Oct 29 21:49:25.056832 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:25.056772    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp\" (UID: \"d220ab29-136b-424f-ab41-af13009c041a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"
Oct 29 21:49:25.057372 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:25.057348    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp\" (UID: \"d220ab29-136b-424f-ab41-af13009c041a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"
Oct 29 21:49:25.082461 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:25.082385    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-922lk\" (UniqueName: \"kubernetes.io/projected/d220ab29-136b-424f-ab41-af13009c041a-kube-api-access-922lk\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp\" (UID: \"d220ab29-136b-424f-ab41-af13009c041a\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"
Oct 29 21:49:25.227595 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:25.227503    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"
Oct 29 21:49:25.564091 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:25.564047    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"]
Oct 29 21:49:25.575553 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:49:25.575498    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd220ab29_136b_424f_ab41_af13009c041a.slice/crio-0ed3d6cc0f37a5fff615531f543953d5156df779358d2570273d0a378ab05c69 WatchSource:0}: Error finding container 0ed3d6cc0f37a5fff615531f543953d5156df779358d2570273d0a378ab05c69: Status 404 returned error can't find the container with id 0ed3d6cc0f37a5fff615531f543953d5156df779358d2570273d0a378ab05c69
Oct 29 21:49:26.278561 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.278451    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp" event={"ID":"d220ab29-136b-424f-ab41-af13009c041a","Type":"ContainerStarted","Data":"0ed3d6cc0f37a5fff615531f543953d5156df779358d2570273d0a378ab05c69"}
Oct 29 21:49:26.855664 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.855064    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"]
Oct 29 21:49:26.862621 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.861389    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:26.876432 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.876396    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b4c9t\""
Oct 29 21:49:26.878081 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.877917    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 29 21:49:26.878081 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.878004    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 29 21:49:26.889558 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.889518    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 29 21:49:26.890898 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.890869    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"]
Oct 29 21:49:26.975191 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.975152    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:26.975401 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.975239    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:26.975401 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.975281    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c28b204-2e05-473f-b027-13d086286e51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:26.975401 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.975333    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:26.975567 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.975417    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:26.975567 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.975492    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:26.975567 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.975522    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c28b204-2e05-473f-b027-13d086286e51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:26.975567 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.975556    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:26.975789 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.975578    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g87s2\" (UniqueName: \"kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-kube-api-access-g87s2\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:26.986461 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.986425    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"]
Oct 29 21:49:26.990763 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:26.990257    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.006055 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.005991    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-tvqqj\""
Oct 29 21:49:27.017619 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.017554    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"]
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.076615    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.076697    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.076805    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c28b204-2e05-473f-b027-13d086286e51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.076855    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b93e523-6093-4484-98c4-ee9762845a2c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.076954    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.077008    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.077034    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.077067    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.077102    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c28b204-2e05-473f-b027-13d086286e51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.077175    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdgvr\" (UniqueName: \"kubernetes.io/projected/1b93e523-6093-4484-98c4-ee9762845a2c-kube-api-access-mdgvr\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.077207    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.077241    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b93e523-6093-4484-98c4-ee9762845a2c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.077280    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.077307    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g87s2\" (UniqueName: \"kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-kube-api-access-g87s2\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.077357    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.077448    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.077487    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.077662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.077515    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b93e523-6093-4484-98c4-ee9762845a2c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.078760 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.078127    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.079200 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.079096    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.080012 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.079706    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c28b204-2e05-473f-b027-13d086286e51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.080012 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.079868    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.080012 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.079951    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.086366 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.086310    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.087494 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.087409    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c28b204-2e05-473f-b027-13d086286e51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.103760 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.103689    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g87s2\" (UniqueName: \"kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-kube-api-access-g87s2\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.103965 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.103921    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-f556s\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.178393 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.178191    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.178393 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.178316    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdgvr\" (UniqueName: \"kubernetes.io/projected/1b93e523-6093-4484-98c4-ee9762845a2c-kube-api-access-mdgvr\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.178393 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.178340    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.178393 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.178361    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b93e523-6093-4484-98c4-ee9762845a2c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.178837 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.178413    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.178837 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.178446    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.178837 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.178473    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b93e523-6093-4484-98c4-ee9762845a2c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.178837 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.178508    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.178837 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.178579    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b93e523-6093-4484-98c4-ee9762845a2c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.182074 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.179870    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b93e523-6093-4484-98c4-ee9762845a2c-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.182074 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.180379    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.182074 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.181035    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.182074 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.181311    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.182074 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.181516    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:27.182074 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.181971    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.182481 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.182411    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b93e523-6093-4484-98c4-ee9762845a2c-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.183955 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.183860    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b93e523-6093-4484-98c4-ee9762845a2c-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.206346 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.206274    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdgvr\" (UniqueName: \"kubernetes.io/projected/1b93e523-6093-4484-98c4-ee9762845a2c-kube-api-access-mdgvr\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.206659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.206614    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b93e523-6093-4484-98c4-ee9762845a2c-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv\" (UID: \"1b93e523-6093-4484-98c4-ee9762845a2c\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.255664 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.255603    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"]
Oct 29 21:49:27.281950 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.281910    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"]
Oct 29 21:49:27.282422 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.282053    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"
Oct 29 21:49:27.310230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.310010    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:27.380905 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.380867    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m\" (UID: \"4a673838-7597-40a2-97c3-5afbc70b6f70\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"
Oct 29 21:49:27.381066 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.380972    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m\" (UID: \"4a673838-7597-40a2-97c3-5afbc70b6f70\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"
Oct 29 21:49:27.381066 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.381034    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whk4x\" (UniqueName: \"kubernetes.io/projected/4a673838-7597-40a2-97c3-5afbc70b6f70-kube-api-access-whk4x\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m\" (UID: \"4a673838-7597-40a2-97c3-5afbc70b6f70\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"
Oct 29 21:49:27.482884 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.482529    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m\" (UID: \"4a673838-7597-40a2-97c3-5afbc70b6f70\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"
Oct 29 21:49:27.482884 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.482590    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whk4x\" (UniqueName: \"kubernetes.io/projected/4a673838-7597-40a2-97c3-5afbc70b6f70-kube-api-access-whk4x\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m\" (UID: \"4a673838-7597-40a2-97c3-5afbc70b6f70\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"
Oct 29 21:49:27.482884 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.482684    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m\" (UID: \"4a673838-7597-40a2-97c3-5afbc70b6f70\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"
Oct 29 21:49:27.483180 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.483074    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m\" (UID: \"4a673838-7597-40a2-97c3-5afbc70b6f70\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"
Oct 29 21:49:27.483745 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.483695    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m\" (UID: \"4a673838-7597-40a2-97c3-5afbc70b6f70\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"
Oct 29 21:49:27.513892 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.513854    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whk4x\" (UniqueName: \"kubernetes.io/projected/4a673838-7597-40a2-97c3-5afbc70b6f70-kube-api-access-whk4x\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m\" (UID: \"4a673838-7597-40a2-97c3-5afbc70b6f70\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"
Oct 29 21:49:27.534682 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.534574    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"]
Oct 29 21:49:27.541485 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:49:27.541447    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c28b204_2e05_473f_b027_13d086286e51.slice/crio-e58eca4c908c07d75ae18f36834145a83f054eba56b3df596686cf9413df6d83 WatchSource:0}: Error finding container e58eca4c908c07d75ae18f36834145a83f054eba56b3df596686cf9413df6d83: Status 404 returned error can't find the container with id e58eca4c908c07d75ae18f36834145a83f054eba56b3df596686cf9413df6d83
Oct 29 21:49:27.609763 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.609724    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"
Oct 29 21:49:27.645859 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.645767    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"]
Oct 29 21:49:27.943600 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:27.942386    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"]
Oct 29 21:49:27.949638 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:49:27.949577    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a673838_7597_40a2_97c3_5afbc70b6f70.slice/crio-4477e7d2d51f036d8d9169f3bfa05162d7da32b3e19ced28e0e6a519ae57b95f WatchSource:0}: Error finding container 4477e7d2d51f036d8d9169f3bfa05162d7da32b3e19ced28e0e6a519ae57b95f: Status 404 returned error can't find the container with id 4477e7d2d51f036d8d9169f3bfa05162d7da32b3e19ced28e0e6a519ae57b95f
Oct 29 21:49:28.302621 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:28.302527    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv" event={"ID":"1b93e523-6093-4484-98c4-ee9762845a2c","Type":"ContainerStarted","Data":"6f9730f93ceaa79302ab5c12ec81ac8a120e09f96a99eb25e0566671b92b8bc3"}
Oct 29 21:49:28.305488 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:28.305272    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m" event={"ID":"4a673838-7597-40a2-97c3-5afbc70b6f70","Type":"ContainerStarted","Data":"4477e7d2d51f036d8d9169f3bfa05162d7da32b3e19ced28e0e6a519ae57b95f"}
Oct 29 21:49:28.308114 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:28.308039    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" event={"ID":"3c28b204-2e05-473f-b027-13d086286e51","Type":"ContainerStarted","Data":"e58eca4c908c07d75ae18f36834145a83f054eba56b3df596686cf9413df6d83"}
Oct 29 21:49:28.435186 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:49:28.435097    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:49:28.827692 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:28.827654    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"]
Oct 29 21:49:28.834308 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:28.833704    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"
Oct 29 21:49:28.862182 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:28.862153    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"]
Oct 29 21:49:28.898968 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:28.898923    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd\" (UID: \"8ff64e39-a46c-4b5c-8371-390722f86031\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"
Oct 29 21:49:28.899127 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:28.899010    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sch5\" (UniqueName: \"kubernetes.io/projected/8ff64e39-a46c-4b5c-8371-390722f86031-kube-api-access-7sch5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd\" (UID: \"8ff64e39-a46c-4b5c-8371-390722f86031\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"
Oct 29 21:49:28.899127 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:28.899104    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd\" (UID: \"8ff64e39-a46c-4b5c-8371-390722f86031\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"
Oct 29 21:49:28.999913 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:28.999873    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd\" (UID: \"8ff64e39-a46c-4b5c-8371-390722f86031\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"
Oct 29 21:49:29.000122 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:28.999996    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd\" (UID: \"8ff64e39-a46c-4b5c-8371-390722f86031\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"
Oct 29 21:49:29.000122 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:29.000068    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sch5\" (UniqueName: \"kubernetes.io/projected/8ff64e39-a46c-4b5c-8371-390722f86031-kube-api-access-7sch5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd\" (UID: \"8ff64e39-a46c-4b5c-8371-390722f86031\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"
Oct 29 21:49:29.000445 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:29.000382    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd\" (UID: \"8ff64e39-a46c-4b5c-8371-390722f86031\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"
Oct 29 21:49:29.000740 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:29.000706    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd\" (UID: \"8ff64e39-a46c-4b5c-8371-390722f86031\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"
Oct 29 21:49:29.032534 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:29.032500    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sch5\" (UniqueName: \"kubernetes.io/projected/8ff64e39-a46c-4b5c-8371-390722f86031-kube-api-access-7sch5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd\" (UID: \"8ff64e39-a46c-4b5c-8371-390722f86031\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"
Oct 29 21:49:29.158405 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:29.158321    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"
Oct 29 21:49:29.495683 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:29.495601    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"]
Oct 29 21:49:30.330732 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:30.330645    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd" event={"ID":"8ff64e39-a46c-4b5c-8371-390722f86031","Type":"ContainerStarted","Data":"16ecfdf91e1cb42449f9674c631b850968b38641ddbd6eb079b430bb83c70d02"}
Oct 29 21:49:31.341872 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:31.341817    7082 generic.go:356] "Generic (PLEG): container finished" podID="4a48c8b5-1dd2-4533-8d64-63843a762d2e" containerID="6821a8d829d5049a2e15ed6542fd1ea9072fcdf87a3372286e604f9cc5fa93c6" exitCode=0
Oct 29 21:49:31.342680 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:31.342644    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs" event={"ID":"4a48c8b5-1dd2-4533-8d64-63843a762d2e","Type":"ContainerDied","Data":"6821a8d829d5049a2e15ed6542fd1ea9072fcdf87a3372286e604f9cc5fa93c6"}
Oct 29 21:49:31.347708 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:31.346980    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s" event={"ID":"33547cfa-0e45-4620-b09c-bf6d98ec4f0a","Type":"ContainerStarted","Data":"7c0890a0ae271bb62322f55fff21847513a7dd7ec9a9ca405389e7517fea1eb0"}
Oct 29 21:49:32.357329 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:32.357291    7082 generic.go:356] "Generic (PLEG): container finished" podID="33547cfa-0e45-4620-b09c-bf6d98ec4f0a" containerID="7c0890a0ae271bb62322f55fff21847513a7dd7ec9a9ca405389e7517fea1eb0" exitCode=0
Oct 29 21:49:32.357851 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:32.357382    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s" event={"ID":"33547cfa-0e45-4620-b09c-bf6d98ec4f0a","Type":"ContainerDied","Data":"7c0890a0ae271bb62322f55fff21847513a7dd7ec9a9ca405389e7517fea1eb0"}
Oct 29 21:49:33.367341 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:33.367302    7082 generic.go:356] "Generic (PLEG): container finished" podID="33547cfa-0e45-4620-b09c-bf6d98ec4f0a" containerID="7676bec4ebacc94ea82e117968bde7cadc5fb926716b7b48966b4faa23be2c3f" exitCode=0
Oct 29 21:49:33.367816 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:33.367382    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s" event={"ID":"33547cfa-0e45-4620-b09c-bf6d98ec4f0a","Type":"ContainerDied","Data":"7676bec4ebacc94ea82e117968bde7cadc5fb926716b7b48966b4faa23be2c3f"}
Oct 29 21:49:33.759041 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:33.758998    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:49:33.759138 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:33.759094    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:49:33.759182 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:33.759139    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:49:34.307664 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:34.307595    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:49:34.307775 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:34.307692    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:49:34.307775 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:34.307725    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:49:34.384389 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:34.380086    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" event={"ID":"3c28b204-2e05-473f-b027-13d086286e51","Type":"ContainerStarted","Data":"f1ff411dd64f982b0603fb68bd3230ba10852f736d1c301d91ebe48d946a438d"}
Oct 29 21:49:34.384815 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:34.384396    7082 generic.go:356] "Generic (PLEG): container finished" podID="d220ab29-136b-424f-ab41-af13009c041a" containerID="2ff83876336f7694b3d35aa1997a67f1df4ed9781e6dfa1a5e3dee7a6d152537" exitCode=0
Oct 29 21:49:34.384815 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:34.384441    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp" event={"ID":"d220ab29-136b-424f-ab41-af13009c041a","Type":"ContainerDied","Data":"2ff83876336f7694b3d35aa1997a67f1df4ed9781e6dfa1a5e3dee7a6d152537"}
Oct 29 21:49:34.488984 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:34.488281    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" podStartSLOduration=2.27648581 podStartE2EDuration="8.48826062s" totalImagesPullingTime="6.21177481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:34.444034496 +0000 UTC m=+146.944076797" watchObservedRunningTime="2026-10-29 21:49:34.48826062 +0000 UTC m=+146.988302919"
Oct 29 21:49:35.182494 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:35.182463    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:35.188727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:35.188471    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:35.397934 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:35.397896    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv" event={"ID":"1b93e523-6093-4484-98c4-ee9762845a2c","Type":"ContainerStarted","Data":"88cc247d438bc843b043dcc02aaef011f5fe663269b6dbbf49f5afd1b517155b"}
Oct 29 21:49:35.398410 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:35.397962    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:35.399117 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:35.399083    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:49:35.445036 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:35.444939    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv" podStartSLOduration=2.793232349 podStartE2EDuration="9.444920734s" totalImagesPullingTime="6.651688385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:35.443996601 +0000 UTC m=+147.944038903" watchObservedRunningTime="2026-10-29 21:49:35.444920734 +0000 UTC m=+147.944963033"
Oct 29 21:49:36.311234 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:36.311197    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:36.316900 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:36.316870    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:36.404190 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:36.404155    7082 generic.go:356] "Generic (PLEG): container finished" podID="1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e" containerID="2c453941a94f8a74e4a20f782a75ba5230d8bc7c7593fc7686162d0cd80b38b0" exitCode=0
Oct 29 21:49:36.404675 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:36.404233    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29" event={"ID":"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e","Type":"ContainerDied","Data":"2c453941a94f8a74e4a20f782a75ba5230d8bc7c7593fc7686162d0cd80b38b0"}
Oct 29 21:49:36.410384 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:36.410355    7082 generic.go:356] "Generic (PLEG): container finished" podID="4a673838-7597-40a2-97c3-5afbc70b6f70" containerID="0e251bd4ec047dfed4a5ab3fa64140c9e4411c84919f2b1defbd8e8fcf0cb5d2" exitCode=0
Oct 29 21:49:36.410553 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:36.410467    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m" event={"ID":"4a673838-7597-40a2-97c3-5afbc70b6f70","Type":"ContainerDied","Data":"0e251bd4ec047dfed4a5ab3fa64140c9e4411c84919f2b1defbd8e8fcf0cb5d2"}
Oct 29 21:49:36.416179 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:36.416151    7082 generic.go:356] "Generic (PLEG): container finished" podID="4a48c8b5-1dd2-4533-8d64-63843a762d2e" containerID="cfae80e7c60c30655a44da4be0f13306228ef365e3dbe2300bef8ef833cac3a5" exitCode=0
Oct 29 21:49:36.417196 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:36.417172    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs" event={"ID":"4a48c8b5-1dd2-4533-8d64-63843a762d2e","Type":"ContainerDied","Data":"cfae80e7c60c30655a44da4be0f13306228ef365e3dbe2300bef8ef833cac3a5"}
Oct 29 21:49:36.418011 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:36.417977    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:36.419054 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:36.419036    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv"
Oct 29 21:49:37.424638 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:37.424600    7082 generic.go:356] "Generic (PLEG): container finished" podID="1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e" containerID="0339e64587011886967b095a9d084a641f0876fa0f7639278476bf77c06d01ea" exitCode=0
Oct 29 21:49:37.425077 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:37.424821    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29" event={"ID":"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e","Type":"ContainerDied","Data":"0339e64587011886967b095a9d084a641f0876fa0f7639278476bf77c06d01ea"}
Oct 29 21:49:37.427854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:37.427827    7082 generic.go:356] "Generic (PLEG): container finished" podID="4a673838-7597-40a2-97c3-5afbc70b6f70" containerID="19aed2044c628a3b89956e6fce722dce4200cce150444c72858d272b5a43157f" exitCode=0
Oct 29 21:49:37.428539 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:37.428510    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m" event={"ID":"4a673838-7597-40a2-97c3-5afbc70b6f70","Type":"ContainerDied","Data":"19aed2044c628a3b89956e6fce722dce4200cce150444c72858d272b5a43157f"}
Oct 29 21:49:37.430707 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:37.430680    7082 generic.go:356] "Generic (PLEG): container finished" podID="8ff64e39-a46c-4b5c-8371-390722f86031" containerID="ad3c61fa77bfd51a6f47b1b18d0349d1ed475dbd083f1409ab5011aa08abddda" exitCode=0
Oct 29 21:49:37.430804 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:37.430764    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd" event={"ID":"8ff64e39-a46c-4b5c-8371-390722f86031","Type":"ContainerDied","Data":"ad3c61fa77bfd51a6f47b1b18d0349d1ed475dbd083f1409ab5011aa08abddda"}
Oct 29 21:49:38.439761 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:38.439714    7082 generic.go:356] "Generic (PLEG): container finished" podID="8ff64e39-a46c-4b5c-8371-390722f86031" containerID="c8048400a91132b7759c696083a613a9d9cc5a1d495284228425947402a3b3cf" exitCode=0
Oct 29 21:49:38.440535 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:38.440508    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd" event={"ID":"8ff64e39-a46c-4b5c-8371-390722f86031","Type":"ContainerDied","Data":"c8048400a91132b7759c696083a613a9d9cc5a1d495284228425947402a3b3cf"}
Oct 29 21:49:38.448199 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:38.448169    7082 generic.go:356] "Generic (PLEG): container finished" podID="d220ab29-136b-424f-ab41-af13009c041a" containerID="4b3d9deb947d3daa2911aafe59c92dc896c32beea3d40f6423f984aa8534697e" exitCode=0
Oct 29 21:49:38.449435 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:38.449326    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp" event={"ID":"d220ab29-136b-424f-ab41-af13009c041a","Type":"ContainerDied","Data":"4b3d9deb947d3daa2911aafe59c92dc896c32beea3d40f6423f984aa8534697e"}
Oct 29 21:49:46.523775 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:46.523744    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd" event={"ID":"8ff64e39-a46c-4b5c-8371-390722f86031","Type":"ContainerStarted","Data":"f9cdcedf6cf9342b60495ac0c86570d8b143507426f06aa36a84976a5ece00c3"}
Oct 29 21:49:46.526927 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:46.526901    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s" event={"ID":"33547cfa-0e45-4620-b09c-bf6d98ec4f0a","Type":"ContainerStarted","Data":"3ec6a717987cece5d23081f51a7580a673adc0958cbfa1be8962885a293588a9"}
Oct 29 21:49:46.568872 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:46.568816    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd" podStartSLOduration=3.168865902 podStartE2EDuration="18.568797043s" totalImagesPullingTime="15.399931141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:46.567803688 +0000 UTC m=+159.067845988" watchObservedRunningTime="2026-10-29 21:49:46.568797043 +0000 UTC m=+159.068839346"
Oct 29 21:49:47.534876 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.534788    7082 generic.go:356] "Generic (PLEG): container finished" podID="8ff64e39-a46c-4b5c-8371-390722f86031" containerID="f9cdcedf6cf9342b60495ac0c86570d8b143507426f06aa36a84976a5ece00c3" exitCode=0
Oct 29 21:49:47.535314 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.534895    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd" event={"ID":"8ff64e39-a46c-4b5c-8371-390722f86031","Type":"ContainerDied","Data":"f9cdcedf6cf9342b60495ac0c86570d8b143507426f06aa36a84976a5ece00c3"}
Oct 29 21:49:47.537742 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.537712    7082 generic.go:356] "Generic (PLEG): container finished" podID="33547cfa-0e45-4620-b09c-bf6d98ec4f0a" containerID="3ec6a717987cece5d23081f51a7580a673adc0958cbfa1be8962885a293588a9" exitCode=0
Oct 29 21:49:47.537875 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.537828    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s" event={"ID":"33547cfa-0e45-4620-b09c-bf6d98ec4f0a","Type":"ContainerDied","Data":"3ec6a717987cece5d23081f51a7580a673adc0958cbfa1be8962885a293588a9"}
Oct 29 21:49:47.540883 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.540854    7082 generic.go:356] "Generic (PLEG): container finished" podID="d220ab29-136b-424f-ab41-af13009c041a" containerID="12b00b00c11e300b3aebea1f2e5d229b40a54560774fa792f9ec8e0fad64dfe2" exitCode=0
Oct 29 21:49:47.541018 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.540926    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp" event={"ID":"d220ab29-136b-424f-ab41-af13009c041a","Type":"ContainerDied","Data":"12b00b00c11e300b3aebea1f2e5d229b40a54560774fa792f9ec8e0fad64dfe2"}
Oct 29 21:49:47.719024 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.718994    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"
Oct 29 21:49:47.814266 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.814231    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-util\" (UniqueName: \"kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-util\") pod \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\" (UID: \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\") "
Oct 29 21:49:47.814448 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.814279    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-bundle\") pod \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\" (UID: \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\") "
Oct 29 21:49:47.814448 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.814392    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-kube-api-access-ntkrn\" (UniqueName: \"kubernetes.io/projected/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-kube-api-access-ntkrn\") pod \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\" (UID: \"33547cfa-0e45-4620-b09c-bf6d98ec4f0a\") "
Oct 29 21:49:47.814978 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.814947    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-bundle" pod "33547cfa-0e45-4620-b09c-bf6d98ec4f0a" (UID: "33547cfa-0e45-4620-b09c-bf6d98ec4f0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:47.817476 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.817440    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-kube-api-access-ntkrn" pod "33547cfa-0e45-4620-b09c-bf6d98ec4f0a" (UID: "33547cfa-0e45-4620-b09c-bf6d98ec4f0a"). InnerVolumeSpecName "kube-api-access-ntkrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:49:47.820579 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.820552    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-util" pod "33547cfa-0e45-4620-b09c-bf6d98ec4f0a" (UID: "33547cfa-0e45-4620-b09c-bf6d98ec4f0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:47.915852 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.915816    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntkrn\" (UniqueName: \"kubernetes.io/projected/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-kube-api-access-ntkrn\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:47.915852 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.915851    7082 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-util\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:47.916109 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:47.915867    7082 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33547cfa-0e45-4620-b09c-bf6d98ec4f0a-bundle\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:48.548485 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.548455    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s"
Oct 29 21:49:48.548485 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.548472    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vdh5s" event={"ID":"33547cfa-0e45-4620-b09c-bf6d98ec4f0a","Type":"ContainerDied","Data":"78b0a038959090c55baac9b3f31a005b7695ab9800b1a6b340b8905890891178"}
Oct 29 21:49:48.548959 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.548522    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78b0a038959090c55baac9b3f31a005b7695ab9800b1a6b340b8905890891178"
Oct 29 21:49:48.553029 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.553002    7082 generic.go:356] "Generic (PLEG): container finished" podID="4a48c8b5-1dd2-4533-8d64-63843a762d2e" containerID="153c1618218682b8c723911a39a9629d572bd4cab438b64c1ce23e3f9a986cce" exitCode=0
Oct 29 21:49:48.553589 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.553565    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs" event={"ID":"4a48c8b5-1dd2-4533-8d64-63843a762d2e","Type":"ContainerDied","Data":"153c1618218682b8c723911a39a9629d572bd4cab438b64c1ce23e3f9a986cce"}
Oct 29 21:49:48.734846 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.734822    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"
Oct 29 21:49:48.762110 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.762086    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"
Oct 29 21:49:48.833543 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.833516    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-util\" (UniqueName: \"kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-util\") pod \"8ff64e39-a46c-4b5c-8371-390722f86031\" (UID: \"8ff64e39-a46c-4b5c-8371-390722f86031\") "
Oct 29 21:49:48.833715 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.833549    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-util\" (UniqueName: \"kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-util\") pod \"d220ab29-136b-424f-ab41-af13009c041a\" (UID: \"d220ab29-136b-424f-ab41-af13009c041a\") "
Oct 29 21:49:48.833715 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.833604    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff64e39-a46c-4b5c-8371-390722f86031-kube-api-access-7sch5\" (UniqueName: \"kubernetes.io/projected/8ff64e39-a46c-4b5c-8371-390722f86031-kube-api-access-7sch5\") pod \"8ff64e39-a46c-4b5c-8371-390722f86031\" (UID: \"8ff64e39-a46c-4b5c-8371-390722f86031\") "
Oct 29 21:49:48.833715 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.833675    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d220ab29-136b-424f-ab41-af13009c041a-kube-api-access-922lk\" (UniqueName: \"kubernetes.io/projected/d220ab29-136b-424f-ab41-af13009c041a-kube-api-access-922lk\") pod \"d220ab29-136b-424f-ab41-af13009c041a\" (UID: \"d220ab29-136b-424f-ab41-af13009c041a\") "
Oct 29 21:49:48.833884 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.833861    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-bundle\") pod \"d220ab29-136b-424f-ab41-af13009c041a\" (UID: \"d220ab29-136b-424f-ab41-af13009c041a\") "
Oct 29 21:49:48.833934 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.833915    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-bundle\") pod \"8ff64e39-a46c-4b5c-8371-390722f86031\" (UID: \"8ff64e39-a46c-4b5c-8371-390722f86031\") "
Oct 29 21:49:48.834505 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.834469    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-bundle" pod "8ff64e39-a46c-4b5c-8371-390722f86031" (UID: "8ff64e39-a46c-4b5c-8371-390722f86031"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:48.834824 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.834753    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-bundle" pod "d220ab29-136b-424f-ab41-af13009c041a" (UID: "d220ab29-136b-424f-ab41-af13009c041a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:48.836017 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.835995    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff64e39-a46c-4b5c-8371-390722f86031-kube-api-access-7sch5" pod "8ff64e39-a46c-4b5c-8371-390722f86031" (UID: "8ff64e39-a46c-4b5c-8371-390722f86031"). InnerVolumeSpecName "kube-api-access-7sch5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:49:48.836221 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.836204    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d220ab29-136b-424f-ab41-af13009c041a-kube-api-access-922lk" pod "d220ab29-136b-424f-ab41-af13009c041a" (UID: "d220ab29-136b-424f-ab41-af13009c041a"). InnerVolumeSpecName "kube-api-access-922lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:49:48.837935 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.837912    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-util" pod "d220ab29-136b-424f-ab41-af13009c041a" (UID: "d220ab29-136b-424f-ab41-af13009c041a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:48.838846 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.838827    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-util" pod "8ff64e39-a46c-4b5c-8371-390722f86031" (UID: "8ff64e39-a46c-4b5c-8371-390722f86031"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:48.934987 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.934916    7082 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-bundle\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:48.934987 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.934943    7082 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-bundle\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:48.934987 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.934951    7082 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ff64e39-a46c-4b5c-8371-390722f86031-util\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:48.934987 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.934958    7082 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d220ab29-136b-424f-ab41-af13009c041a-util\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:48.934987 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.934966    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sch5\" (UniqueName: \"kubernetes.io/projected/8ff64e39-a46c-4b5c-8371-390722f86031-kube-api-access-7sch5\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:48.934987 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:48.934976    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-922lk\" (UniqueName: \"kubernetes.io/projected/d220ab29-136b-424f-ab41-af13009c041a-kube-api-access-922lk\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:49.559370 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.559340    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd"
Oct 29 21:49:49.559799 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.559341    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lgpjd" event={"ID":"8ff64e39-a46c-4b5c-8371-390722f86031","Type":"ContainerDied","Data":"16ecfdf91e1cb42449f9674c631b850968b38641ddbd6eb079b430bb83c70d02"}
Oct 29 21:49:49.559799 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.559502    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16ecfdf91e1cb42449f9674c631b850968b38641ddbd6eb079b430bb83c70d02"
Oct 29 21:49:49.561601 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.561575    7082 generic.go:356] "Generic (PLEG): container finished" podID="1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e" containerID="932ff05a683de5eb043fbe3b86bf42b46fc725d3530a1d43ad7f3232744c94d1" exitCode=0
Oct 29 21:49:49.561751 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.561654    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29" event={"ID":"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e","Type":"ContainerDied","Data":"932ff05a683de5eb043fbe3b86bf42b46fc725d3530a1d43ad7f3232744c94d1"}
Oct 29 21:49:49.563447 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.563428    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp"
Oct 29 21:49:49.563572 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.563429    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdc9pp" event={"ID":"d220ab29-136b-424f-ab41-af13009c041a","Type":"ContainerDied","Data":"0ed3d6cc0f37a5fff615531f543953d5156df779358d2570273d0a378ab05c69"}
Oct 29 21:49:49.563572 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.563535    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ed3d6cc0f37a5fff615531f543953d5156df779358d2570273d0a378ab05c69"
Oct 29 21:49:49.565702 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.565683    7082 generic.go:356] "Generic (PLEG): container finished" podID="4a673838-7597-40a2-97c3-5afbc70b6f70" containerID="ce9beace394fc2defb2cc95d6f2581639d0d28cdec745a4b57a867e682b93bf8" exitCode=0
Oct 29 21:49:49.565790 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.565763    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m" event={"ID":"4a673838-7597-40a2-97c3-5afbc70b6f70","Type":"ContainerDied","Data":"ce9beace394fc2defb2cc95d6f2581639d0d28cdec745a4b57a867e682b93bf8"}
Oct 29 21:49:49.728863 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.728831    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"
Oct 29 21:49:49.843964 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.843870    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a48c8b5-1dd2-4533-8d64-63843a762d2e-kube-api-access-695m4\" (UniqueName: \"kubernetes.io/projected/4a48c8b5-1dd2-4533-8d64-63843a762d2e-kube-api-access-695m4\") pod \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\" (UID: \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\") "
Oct 29 21:49:49.843964 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.843945    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-util\" (UniqueName: \"kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-util\") pod \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\" (UID: \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\") "
Oct 29 21:49:49.844208 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.844028    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-bundle\") pod \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\" (UID: \"4a48c8b5-1dd2-4533-8d64-63843a762d2e\") "
Oct 29 21:49:49.844380 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.844357    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-bundle" pod "4a48c8b5-1dd2-4533-8d64-63843a762d2e" (UID: "4a48c8b5-1dd2-4533-8d64-63843a762d2e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:49.846164 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.846142    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a48c8b5-1dd2-4533-8d64-63843a762d2e-kube-api-access-695m4" pod "4a48c8b5-1dd2-4533-8d64-63843a762d2e" (UID: "4a48c8b5-1dd2-4533-8d64-63843a762d2e"). InnerVolumeSpecName "kube-api-access-695m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:49:49.849675 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.849650    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-util" pod "4a48c8b5-1dd2-4533-8d64-63843a762d2e" (UID: "4a48c8b5-1dd2-4533-8d64-63843a762d2e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:49.945647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.945581    7082 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-bundle\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:49.945647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.945615    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-695m4\" (UniqueName: \"kubernetes.io/projected/4a48c8b5-1dd2-4533-8d64-63843a762d2e-kube-api-access-695m4\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:49.945647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:49.945653    7082 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a48c8b5-1dd2-4533-8d64-63843a762d2e-util\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:50.571925 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.571894    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs"
Oct 29 21:49:50.571925 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.571913    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64425srs" event={"ID":"4a48c8b5-1dd2-4533-8d64-63843a762d2e","Type":"ContainerDied","Data":"f15e8d9167b3b2c1cea2b89b22272020ca346a51314cca2bea02255e43b45b2e"}
Oct 29 21:49:50.572464 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.571958    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f15e8d9167b3b2c1cea2b89b22272020ca346a51314cca2bea02255e43b45b2e"
Oct 29 21:49:50.741897 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.741867    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"
Oct 29 21:49:50.757957 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.757930    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"
Oct 29 21:49:50.855946 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.855860    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-util\" (UniqueName: \"kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-util\") pod \"4a673838-7597-40a2-97c3-5afbc70b6f70\" (UID: \"4a673838-7597-40a2-97c3-5afbc70b6f70\") "
Oct 29 21:49:50.856118 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.855946    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-kube-api-access-5j6mp\" (UniqueName: \"kubernetes.io/projected/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-kube-api-access-5j6mp\") pod \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\" (UID: \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\") "
Oct 29 21:49:50.856118 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.855981    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-bundle\") pod \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\" (UID: \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\") "
Oct 29 21:49:50.856118 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.856044    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a673838-7597-40a2-97c3-5afbc70b6f70-kube-api-access-whk4x\" (UniqueName: \"kubernetes.io/projected/4a673838-7597-40a2-97c3-5afbc70b6f70-kube-api-access-whk4x\") pod \"4a673838-7597-40a2-97c3-5afbc70b6f70\" (UID: \"4a673838-7597-40a2-97c3-5afbc70b6f70\") "
Oct 29 21:49:50.856118 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.856070    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-bundle\") pod \"4a673838-7597-40a2-97c3-5afbc70b6f70\" (UID: \"4a673838-7597-40a2-97c3-5afbc70b6f70\") "
Oct 29 21:49:50.856118 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.856104    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-util\" (UniqueName: \"kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-util\") pod \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\" (UID: \"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e\") "
Oct 29 21:49:50.856581 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.856511    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-bundle" pod "4a673838-7597-40a2-97c3-5afbc70b6f70" (UID: "4a673838-7597-40a2-97c3-5afbc70b6f70"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:50.856860 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.856833    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-bundle" pod "1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e" (UID: "1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:50.858836 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.858722    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-kube-api-access-5j6mp" pod "1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e" (UID: "1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e"). InnerVolumeSpecName "kube-api-access-5j6mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:49:50.858836 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.858815    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a673838-7597-40a2-97c3-5afbc70b6f70-kube-api-access-whk4x" pod "4a673838-7597-40a2-97c3-5afbc70b6f70" (UID: "4a673838-7597-40a2-97c3-5afbc70b6f70"). InnerVolumeSpecName "kube-api-access-whk4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:49:50.861227 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.861205    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-util" pod "1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e" (UID: "1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:50.861877 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.861852    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-util" pod "4a673838-7597-40a2-97c3-5afbc70b6f70" (UID: "4a673838-7597-40a2-97c3-5afbc70b6f70"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:50.957033 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.956997    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j6mp\" (UniqueName: \"kubernetes.io/projected/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-kube-api-access-5j6mp\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:50.957033 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.957024    7082 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-bundle\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:50.957033 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.957034    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whk4x\" (UniqueName: \"kubernetes.io/projected/4a673838-7597-40a2-97c3-5afbc70b6f70-kube-api-access-whk4x\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:50.957033 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.957043    7082 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-bundle\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:50.957287 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.957051    7082 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e-util\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:50.957287 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:50.957059    7082 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a673838-7597-40a2-97c3-5afbc70b6f70-util\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:49:51.388776 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.388740    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"]
Oct 29 21:49:51.389070 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389058    7082 cpu_manager.go:414] "containerMap: removing container" podUID="33547cfa-0e45-4620-b09c-bf6d98ec4f0a" containerName="util"
Oct 29 21:49:51.389129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389071    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33547cfa-0e45-4620-b09c-bf6d98ec4f0a" containerName="util"
Oct 29 21:49:51.389129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389082    7082 cpu_manager.go:414] "containerMap: removing container" podUID="8ff64e39-a46c-4b5c-8371-390722f86031" containerName="pull"
Oct 29 21:49:51.389129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389086    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff64e39-a46c-4b5c-8371-390722f86031" containerName="pull"
Oct 29 21:49:51.389129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389094    7082 cpu_manager.go:414] "containerMap: removing container" podUID="4a48c8b5-1dd2-4533-8d64-63843a762d2e" containerName="util"
Oct 29 21:49:51.389129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389099    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a48c8b5-1dd2-4533-8d64-63843a762d2e" containerName="util"
Oct 29 21:49:51.389129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389107    7082 cpu_manager.go:414] "containerMap: removing container" podUID="33547cfa-0e45-4620-b09c-bf6d98ec4f0a" containerName="pull"
Oct 29 21:49:51.389129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389113    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33547cfa-0e45-4620-b09c-bf6d98ec4f0a" containerName="pull"
Oct 29 21:49:51.389129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389119    7082 cpu_manager.go:414] "containerMap: removing container" podUID="33547cfa-0e45-4620-b09c-bf6d98ec4f0a" containerName="extract"
Oct 29 21:49:51.389129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389123    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33547cfa-0e45-4620-b09c-bf6d98ec4f0a" containerName="extract"
Oct 29 21:49:51.389129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389128    7082 cpu_manager.go:414] "containerMap: removing container" podUID="4a48c8b5-1dd2-4533-8d64-63843a762d2e" containerName="extract"
Oct 29 21:49:51.389129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389132    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a48c8b5-1dd2-4533-8d64-63843a762d2e" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389139    7082 cpu_manager.go:414] "containerMap: removing container" podUID="d220ab29-136b-424f-ab41-af13009c041a" containerName="util"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389143    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d220ab29-136b-424f-ab41-af13009c041a" containerName="util"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389149    7082 cpu_manager.go:414] "containerMap: removing container" podUID="8ff64e39-a46c-4b5c-8371-390722f86031" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389153    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff64e39-a46c-4b5c-8371-390722f86031" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389162    7082 cpu_manager.go:414] "containerMap: removing container" podUID="1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389166    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389178    7082 cpu_manager.go:414] "containerMap: removing container" podUID="4a673838-7597-40a2-97c3-5afbc70b6f70" containerName="util"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389182    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a673838-7597-40a2-97c3-5afbc70b6f70" containerName="util"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389188    7082 cpu_manager.go:414] "containerMap: removing container" podUID="d220ab29-136b-424f-ab41-af13009c041a" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389192    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d220ab29-136b-424f-ab41-af13009c041a" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389198    7082 cpu_manager.go:414] "containerMap: removing container" podUID="4a673838-7597-40a2-97c3-5afbc70b6f70" containerName="pull"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389202    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a673838-7597-40a2-97c3-5afbc70b6f70" containerName="pull"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389213    7082 cpu_manager.go:414] "containerMap: removing container" podUID="4a48c8b5-1dd2-4533-8d64-63843a762d2e" containerName="pull"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389216    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a48c8b5-1dd2-4533-8d64-63843a762d2e" containerName="pull"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389223    7082 cpu_manager.go:414] "containerMap: removing container" podUID="4a673838-7597-40a2-97c3-5afbc70b6f70" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389226    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a673838-7597-40a2-97c3-5afbc70b6f70" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389233    7082 cpu_manager.go:414] "containerMap: removing container" podUID="1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e" containerName="util"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389238    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e" containerName="util"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389244    7082 cpu_manager.go:414] "containerMap: removing container" podUID="d220ab29-136b-424f-ab41-af13009c041a" containerName="pull"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389247    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d220ab29-136b-424f-ab41-af13009c041a" containerName="pull"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389256    7082 cpu_manager.go:414] "containerMap: removing container" podUID="1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e" containerName="pull"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389260    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e" containerName="pull"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389267    7082 cpu_manager.go:414] "containerMap: removing container" podUID="8ff64e39-a46c-4b5c-8371-390722f86031" containerName="util"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389271    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff64e39-a46c-4b5c-8371-390722f86031" containerName="util"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389314    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a48c8b5-1dd2-4533-8d64-63843a762d2e" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389320    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="33547cfa-0e45-4620-b09c-bf6d98ec4f0a" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389326    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff64e39-a46c-4b5c-8371-390722f86031" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389332    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389338    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="d220ab29-136b-424f-ab41-af13009c041a" containerName="extract"
Oct 29 21:49:51.389414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.389344    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a673838-7597-40a2-97c3-5afbc70b6f70" containerName="extract"
Oct 29 21:49:51.429323 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.429294    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.453956 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.453923    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 29 21:49:51.454093 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.454001    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 29 21:49:51.454093 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.454065    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 29 21:49:51.454093 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.454074    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 29 21:49:51.454217 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.454121    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 29 21:49:51.464864 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.464843    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-6rlhk\""
Oct 29 21:49:51.473641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.473601    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"]
Oct 29 21:49:51.561139 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.561101    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4b42b80-df94-4d12-80cb-5814f733d918-tmp\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.561310 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.561159    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq6gr\" (UniqueName: \"kubernetes.io/projected/f4b42b80-df94-4d12-80cb-5814f733d918-kube-api-access-qq6gr\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.561310 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.561236    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f4b42b80-df94-4d12-80cb-5814f733d918-manifests\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.561384 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.561337    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f4b42b80-df94-4d12-80cb-5814f733d918-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.561384 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.561364    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4b42b80-df94-4d12-80cb-5814f733d918-webhook-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.561444 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.561426    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4b42b80-df94-4d12-80cb-5814f733d918-metrics-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.578937 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.578915    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29"
Oct 29 21:49:51.579327 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.578912    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/fdb36248658848c45b2cd30265e4aa38f5747d5521cc1af4f2a8142ceffxr29" event={"ID":"1f183d68-fc07-43a1-8af2-d6e7bd2c3c7e","Type":"ContainerDied","Data":"b4c32b12876ba5a91fca52e9cbcc470984e15f6da5c82b30baff8f72d16be842"}
Oct 29 21:49:51.579327 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.579066    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4c32b12876ba5a91fca52e9cbcc470984e15f6da5c82b30baff8f72d16be842"
Oct 29 21:49:51.580856 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.580835    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m" event={"ID":"4a673838-7597-40a2-97c3-5afbc70b6f70","Type":"ContainerDied","Data":"4477e7d2d51f036d8d9169f3bfa05162d7da32b3e19ced28e0e6a519ae57b95f"}
Oct 29 21:49:51.580941 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.580876    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4477e7d2d51f036d8d9169f3bfa05162d7da32b3e19ced28e0e6a519ae57b95f"
Oct 29 21:49:51.580941 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.580882    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tk59m"
Oct 29 21:49:51.662595 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.662515    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4b42b80-df94-4d12-80cb-5814f733d918-tmp\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.662595 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.662590    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq6gr\" (UniqueName: \"kubernetes.io/projected/f4b42b80-df94-4d12-80cb-5814f733d918-kube-api-access-qq6gr\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.662837 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.662611    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f4b42b80-df94-4d12-80cb-5814f733d918-manifests\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.662837 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.662657    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f4b42b80-df94-4d12-80cb-5814f733d918-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.662837 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.662674    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4b42b80-df94-4d12-80cb-5814f733d918-webhook-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.662837 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.662697    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4b42b80-df94-4d12-80cb-5814f733d918-metrics-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.663092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.663068    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f4b42b80-df94-4d12-80cb-5814f733d918-manifests\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.665001 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.664972    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4b42b80-df94-4d12-80cb-5814f733d918-tmp\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.665246 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.665228    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f4b42b80-df94-4d12-80cb-5814f733d918-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.665296 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.665255    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f4b42b80-df94-4d12-80cb-5814f733d918-metrics-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.665333 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.665313    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4b42b80-df94-4d12-80cb-5814f733d918-webhook-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.686279 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.686256    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq6gr\" (UniqueName: \"kubernetes.io/projected/f4b42b80-df94-4d12-80cb-5814f733d918-kube-api-access-qq6gr\") pod \"opendatahub-operator-controller-manager-59b57f6f78-t48hs\" (UID: \"f4b42b80-df94-4d12-80cb-5814f733d918\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:51.772194 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:51.772158    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:52.119329 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:52.119190    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"]
Oct 29 21:49:52.121996 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:49:52.121965    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b42b80_df94_4d12_80cb_5814f733d918.slice/crio-f76c1f3b8d13b1cc53011db86405c8ab0b5506fd0df91fabe2d802ca1f380edf WatchSource:0}: Error finding container f76c1f3b8d13b1cc53011db86405c8ab0b5506fd0df91fabe2d802ca1f380edf: Status 404 returned error can't find the container with id f76c1f3b8d13b1cc53011db86405c8ab0b5506fd0df91fabe2d802ca1f380edf
Oct 29 21:49:52.586100 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:52.586061    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs" event={"ID":"f4b42b80-df94-4d12-80cb-5814f733d918","Type":"ContainerStarted","Data":"f76c1f3b8d13b1cc53011db86405c8ab0b5506fd0df91fabe2d802ca1f380edf"}
Oct 29 21:49:55.603198 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:55.603163    7082 generic.go:356] "Generic (PLEG): container finished" podID="f4b42b80-df94-4d12-80cb-5814f733d918" containerID="51a7189a89cec59cc00819c717c60d2bb5246aa53fe49af171a4bb027ee6fd5a" exitCode=0
Oct 29 21:49:55.603710 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:55.603255    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs" event={"ID":"f4b42b80-df94-4d12-80cb-5814f733d918","Type":"ContainerDied","Data":"51a7189a89cec59cc00819c717c60d2bb5246aa53fe49af171a4bb027ee6fd5a"}
Oct 29 21:49:56.618675 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:56.618614    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs" event={"ID":"f4b42b80-df94-4d12-80cb-5814f733d918","Type":"ContainerStarted","Data":"1b1ce3527cb87d2cb6197f058b48c9bd38090af9b9c969e48a2778ec15377f45"}
Oct 29 21:49:56.619179 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:56.618911    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:49:56.709561 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:56.709502    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs" podStartSLOduration=1.9868538949999999 podStartE2EDuration="5.709483164s" totalImagesPullingTime="3.243905497s" totalInitContainerRuntime="478.723772ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:56.696558994 +0000 UTC m=+169.196601294" watchObservedRunningTime="2026-10-29 21:49:56.709483164 +0000 UTC m=+169.209525464"
Oct 29 21:49:58.411074 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:49:58.411040    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:49:59.100208 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.100168    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6"]
Oct 29 21:49:59.103742 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.103720    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6"
Oct 29 21:49:59.132686 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.132656    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:49:59.136070 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.136042    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 29 21:49:59.136217 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.136085    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-bp79k\""
Oct 29 21:49:59.137295 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.137272    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6"]
Oct 29 21:49:59.231611 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.231575    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bea0920-e655-43aa-8d9d-39f050650de8-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cncs6\" (UID: \"3bea0920-e655-43aa-8d9d-39f050650de8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6"
Oct 29 21:49:59.231795 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.231698    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdd6z\" (UniqueName: \"kubernetes.io/projected/3bea0920-e655-43aa-8d9d-39f050650de8-kube-api-access-qdd6z\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cncs6\" (UID: \"3bea0920-e655-43aa-8d9d-39f050650de8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6"
Oct 29 21:49:59.332528 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.332487    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdd6z\" (UniqueName: \"kubernetes.io/projected/3bea0920-e655-43aa-8d9d-39f050650de8-kube-api-access-qdd6z\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cncs6\" (UID: \"3bea0920-e655-43aa-8d9d-39f050650de8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6"
Oct 29 21:49:59.332709 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.332595    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bea0920-e655-43aa-8d9d-39f050650de8-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cncs6\" (UID: \"3bea0920-e655-43aa-8d9d-39f050650de8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6"
Oct 29 21:49:59.333011 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.332993    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bea0920-e655-43aa-8d9d-39f050650de8-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cncs6\" (UID: \"3bea0920-e655-43aa-8d9d-39f050650de8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6"
Oct 29 21:49:59.357561 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.357492    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdd6z\" (UniqueName: \"kubernetes.io/projected/3bea0920-e655-43aa-8d9d-39f050650de8-kube-api-access-qdd6z\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cncs6\" (UID: \"3bea0920-e655-43aa-8d9d-39f050650de8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6"
Oct 29 21:49:59.415544 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.415511    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6"
Oct 29 21:49:59.752874 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:49:59.752795    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6"]
Oct 29 21:49:59.756159 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:49:59.756135    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bea0920_e655_43aa_8d9d_39f050650de8.slice/crio-eea85b7c73053009c9c5005afe8dd7b5f07fb15be0d0b67aeb4378e9d689575e WatchSource:0}: Error finding container eea85b7c73053009c9c5005afe8dd7b5f07fb15be0d0b67aeb4378e9d689575e: Status 404 returned error can't find the container with id eea85b7c73053009c9c5005afe8dd7b5f07fb15be0d0b67aeb4378e9d689575e
Oct 29 21:50:00.637480 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:00.637446    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6" event={"ID":"3bea0920-e655-43aa-8d9d-39f050650de8","Type":"ContainerStarted","Data":"eea85b7c73053009c9c5005afe8dd7b5f07fb15be0d0b67aeb4378e9d689575e"}
Oct 29 21:50:01.643042 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:01.643009    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6" event={"ID":"3bea0920-e655-43aa-8d9d-39f050650de8","Type":"ContainerStarted","Data":"9d3a54b492e9073489306aed2e2d85c8bd56b7c317f28580fd682492b9e60272"}
Oct 29 21:50:01.715919 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:01.715742    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cncs6" podStartSLOduration=1.277629129 podStartE2EDuration="2.715722891s" totalImagesPullingTime="1.438093762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:01.702546162 +0000 UTC m=+174.202588461" watchObservedRunningTime="2026-10-29 21:50:01.715722891 +0000 UTC m=+174.215765193"
Oct 29 21:50:02.879910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:02.879878    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c"]
Oct 29 21:50:02.883469 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:02.883449    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c"
Oct 29 21:50:02.912523 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:02.912496    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 29 21:50:02.912523 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:02.912498    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-sv7kb\""
Oct 29 21:50:02.912771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:02.912498    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:50:02.914851 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:02.914829    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c"]
Oct 29 21:50:02.972186 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:02.972154    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94l7r\" (UniqueName: \"kubernetes.io/projected/0da08d8f-8a7c-44f2-8206-bbf7fdb13685-kube-api-access-94l7r\") pod \"jobset-operator-674cd59dc5-sdx9c\" (UID: \"0da08d8f-8a7c-44f2-8206-bbf7fdb13685\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c"
Oct 29 21:50:02.972186 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:02.972197    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0da08d8f-8a7c-44f2-8206-bbf7fdb13685-tmp\") pod \"jobset-operator-674cd59dc5-sdx9c\" (UID: \"0da08d8f-8a7c-44f2-8206-bbf7fdb13685\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c"
Oct 29 21:50:03.073642 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:03.073598    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94l7r\" (UniqueName: \"kubernetes.io/projected/0da08d8f-8a7c-44f2-8206-bbf7fdb13685-kube-api-access-94l7r\") pod \"jobset-operator-674cd59dc5-sdx9c\" (UID: \"0da08d8f-8a7c-44f2-8206-bbf7fdb13685\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c"
Oct 29 21:50:03.073824 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:03.073656    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0da08d8f-8a7c-44f2-8206-bbf7fdb13685-tmp\") pod \"jobset-operator-674cd59dc5-sdx9c\" (UID: \"0da08d8f-8a7c-44f2-8206-bbf7fdb13685\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c"
Oct 29 21:50:03.074065 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:03.074048    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0da08d8f-8a7c-44f2-8206-bbf7fdb13685-tmp\") pod \"jobset-operator-674cd59dc5-sdx9c\" (UID: \"0da08d8f-8a7c-44f2-8206-bbf7fdb13685\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c"
Oct 29 21:50:03.104975 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:03.104948    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94l7r\" (UniqueName: \"kubernetes.io/projected/0da08d8f-8a7c-44f2-8206-bbf7fdb13685-kube-api-access-94l7r\") pod \"jobset-operator-674cd59dc5-sdx9c\" (UID: \"0da08d8f-8a7c-44f2-8206-bbf7fdb13685\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c"
Oct 29 21:50:03.205237 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:03.205159    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c"
Oct 29 21:50:03.519535 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:03.519463    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c"]
Oct 29 21:50:03.523781 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:50:03.523755    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0da08d8f_8a7c_44f2_8206_bbf7fdb13685.slice/crio-fc419769ec52227b538e95f2af1866468911967e29b84ffdfe6bbb95bfa36baa WatchSource:0}: Error finding container fc419769ec52227b538e95f2af1866468911967e29b84ffdfe6bbb95bfa36baa: Status 404 returned error can't find the container with id fc419769ec52227b538e95f2af1866468911967e29b84ffdfe6bbb95bfa36baa
Oct 29 21:50:03.653532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:03.653501    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c" event={"ID":"0da08d8f-8a7c-44f2-8206-bbf7fdb13685","Type":"ContainerStarted","Data":"fc419769ec52227b538e95f2af1866468911967e29b84ffdfe6bbb95bfa36baa"}
Oct 29 21:50:05.665861 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:05.665818    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c" event={"ID":"0da08d8f-8a7c-44f2-8206-bbf7fdb13685","Type":"ContainerStarted","Data":"03b3ea42a59098e2d57086fd4e5ae2c43d2ba1eab47ddcf3c641932cae450699"}
Oct 29 21:50:05.753073 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:05.753023    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sdx9c" podStartSLOduration=1.742109768 podStartE2EDuration="3.753006353s" totalImagesPullingTime="2.010896585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:05.751579803 +0000 UTC m=+178.251622103" watchObservedRunningTime="2026-10-29 21:50:05.753006353 +0000 UTC m=+178.253048651"
Oct 29 21:50:06.325979 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.325946    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j"]
Oct 29 21:50:06.329499 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.329481    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j"
Oct 29 21:50:06.344595 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.344577    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sft9w\""
Oct 29 21:50:06.350781 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.350759    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j"]
Oct 29 21:50:06.377906 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.377882    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 29 21:50:06.378041 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.377940    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 29 21:50:06.508577 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.508548    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfsmg\" (UniqueName: \"kubernetes.io/projected/1aadf369-2748-40a7-9990-6277f6aa995f-kube-api-access-pfsmg\") pod \"cert-manager-cainjector-5c57bbd4fc-swf6j\" (UID: \"1aadf369-2748-40a7-9990-6277f6aa995f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j"
Oct 29 21:50:06.508739 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.508581    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1aadf369-2748-40a7-9990-6277f6aa995f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-swf6j\" (UID: \"1aadf369-2748-40a7-9990-6277f6aa995f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j"
Oct 29 21:50:06.609754 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.609689    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfsmg\" (UniqueName: \"kubernetes.io/projected/1aadf369-2748-40a7-9990-6277f6aa995f-kube-api-access-pfsmg\") pod \"cert-manager-cainjector-5c57bbd4fc-swf6j\" (UID: \"1aadf369-2748-40a7-9990-6277f6aa995f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j"
Oct 29 21:50:06.609754 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.609722    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1aadf369-2748-40a7-9990-6277f6aa995f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-swf6j\" (UID: \"1aadf369-2748-40a7-9990-6277f6aa995f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j"
Oct 29 21:50:06.638886 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.638864    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1aadf369-2748-40a7-9990-6277f6aa995f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-swf6j\" (UID: \"1aadf369-2748-40a7-9990-6277f6aa995f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j"
Oct 29 21:50:06.638954 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.638931    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfsmg\" (UniqueName: \"kubernetes.io/projected/1aadf369-2748-40a7-9990-6277f6aa995f-kube-api-access-pfsmg\") pod \"cert-manager-cainjector-5c57bbd4fc-swf6j\" (UID: \"1aadf369-2748-40a7-9990-6277f6aa995f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j"
Oct 29 21:50:06.640674 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.640650    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j"
Oct 29 21:50:06.942811 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:06.942667    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j"]
Oct 29 21:50:06.945444 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:50:06.945413    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1aadf369_2748_40a7_9990_6277f6aa995f.slice/crio-2ae091f2479e31b01bedddad2ac7386af5409fbbaeaa40a23590dbbd66b57d57 WatchSource:0}: Error finding container 2ae091f2479e31b01bedddad2ac7386af5409fbbaeaa40a23590dbbd66b57d57: Status 404 returned error can't find the container with id 2ae091f2479e31b01bedddad2ac7386af5409fbbaeaa40a23590dbbd66b57d57
Oct 29 21:50:07.629538 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:07.629513    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 21:50:07.675697 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:07.675664    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j" event={"ID":"1aadf369-2748-40a7-9990-6277f6aa995f","Type":"ContainerStarted","Data":"2ae091f2479e31b01bedddad2ac7386af5409fbbaeaa40a23590dbbd66b57d57"}
Oct 29 21:50:10.692891 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:10.692841    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j" event={"ID":"1aadf369-2748-40a7-9990-6277f6aa995f","Type":"ContainerStarted","Data":"4ffbfd93ef278e3f16800b19516f6807ad85972222ccecf64a6e455a8995d984"}
Oct 29 21:50:10.736092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:10.736031    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-swf6j" podStartSLOduration=1.8171006699999999 podStartE2EDuration="4.736005412s" totalImagesPullingTime="2.918904742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:10.734821226 +0000 UTC m=+183.234863526" watchObservedRunningTime="2026-10-29 21:50:10.736005412 +0000 UTC m=+183.236047711"
Oct 29 21:50:13.851826 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:13.851797    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8"]
Oct 29 21:50:13.880375 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:13.880350    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8"]
Oct 29 21:50:13.880525 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:13.880449    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8"
Oct 29 21:50:13.895053 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:13.895031    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mqxxv\""
Oct 29 21:50:13.896369 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:13.896351    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 29 21:50:13.896606 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:13.896592    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 29 21:50:13.974717 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:13.974678    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cht7z\" (UniqueName: \"kubernetes.io/projected/b8178c0e-7643-499f-9dc0-e04329809348-kube-api-access-cht7z\") pod \"kuadrant-operator-controller-manager-6b575997cc-zthg8\" (UID: \"b8178c0e-7643-499f-9dc0-e04329809348\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8"
Oct 29 21:50:14.076032 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:14.075992    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cht7z\" (UniqueName: \"kubernetes.io/projected/b8178c0e-7643-499f-9dc0-e04329809348-kube-api-access-cht7z\") pod \"kuadrant-operator-controller-manager-6b575997cc-zthg8\" (UID: \"b8178c0e-7643-499f-9dc0-e04329809348\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8"
Oct 29 21:50:14.100112 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:14.100086    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cht7z\" (UniqueName: \"kubernetes.io/projected/b8178c0e-7643-499f-9dc0-e04329809348-kube-api-access-cht7z\") pod \"kuadrant-operator-controller-manager-6b575997cc-zthg8\" (UID: \"b8178c0e-7643-499f-9dc0-e04329809348\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8"
Oct 29 21:50:14.204848 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:14.204775    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8"
Oct 29 21:50:14.556769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:14.556742    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8"]
Oct 29 21:50:14.559197 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:50:14.559169    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8178c0e_7643_499f_9dc0_e04329809348.slice/crio-d4fbdb970dd3aaefa75cf40cfde4eef2fff61ef88e8ee2f32ab204eac0fc2136 WatchSource:0}: Error finding container d4fbdb970dd3aaefa75cf40cfde4eef2fff61ef88e8ee2f32ab204eac0fc2136: Status 404 returned error can't find the container with id d4fbdb970dd3aaefa75cf40cfde4eef2fff61ef88e8ee2f32ab204eac0fc2136
Oct 29 21:50:14.714253 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:14.714213    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8" event={"ID":"b8178c0e-7643-499f-9dc0-e04329809348","Type":"ContainerStarted","Data":"d4fbdb970dd3aaefa75cf40cfde4eef2fff61ef88e8ee2f32ab204eac0fc2136"}
Oct 29 21:50:21.750959 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:21.750919    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8" event={"ID":"b8178c0e-7643-499f-9dc0-e04329809348","Type":"ContainerStarted","Data":"adeeace6427d775b3c7541202cba74c1947e3885f9d34f2149cd465885c6d6b7"}
Oct 29 21:50:21.751441 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:21.751049    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8"
Oct 29 21:50:21.844577 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:21.844528    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8" podStartSLOduration=2.340724704 podStartE2EDuration="8.844513483s" totalImagesPullingTime="6.503788779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:21.830661307 +0000 UTC m=+194.330703603" watchObservedRunningTime="2026-10-29 21:50:21.844513483 +0000 UTC m=+194.344555847"
Oct 29 21:50:23.133378 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.133333    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"]
Oct 29 21:50:23.137237 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.137215    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"
Oct 29 21:50:23.151052 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.151024    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e810e5c-e197-4b47-b906-95e8b88f5586-tls-certs\") pod \"kube-auth-proxy-64545848b5-ffqsz\" (UID: \"4e810e5c-e197-4b47-b906-95e8b88f5586\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"
Oct 29 21:50:23.151174 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.151068    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hqq4\" (UniqueName: \"kubernetes.io/projected/4e810e5c-e197-4b47-b906-95e8b88f5586-kube-api-access-9hqq4\") pod \"kube-auth-proxy-64545848b5-ffqsz\" (UID: \"4e810e5c-e197-4b47-b906-95e8b88f5586\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"
Oct 29 21:50:23.151214 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.151163    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e810e5c-e197-4b47-b906-95e8b88f5586-tmp\") pod \"kube-auth-proxy-64545848b5-ffqsz\" (UID: \"4e810e5c-e197-4b47-b906-95e8b88f5586\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"
Oct 29 21:50:23.151832 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.151816    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4n74v\""
Oct 29 21:50:23.155762 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.155741    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 29 21:50:23.165652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.165601    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"]
Oct 29 21:50:23.241267 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.241233    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 29 21:50:23.251610 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.251580    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e810e5c-e197-4b47-b906-95e8b88f5586-tmp\") pod \"kube-auth-proxy-64545848b5-ffqsz\" (UID: \"4e810e5c-e197-4b47-b906-95e8b88f5586\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"
Oct 29 21:50:23.251814 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.251683    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e810e5c-e197-4b47-b906-95e8b88f5586-tls-certs\") pod \"kube-auth-proxy-64545848b5-ffqsz\" (UID: \"4e810e5c-e197-4b47-b906-95e8b88f5586\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"
Oct 29 21:50:23.251814 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.251716    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hqq4\" (UniqueName: \"kubernetes.io/projected/4e810e5c-e197-4b47-b906-95e8b88f5586-kube-api-access-9hqq4\") pod \"kube-auth-proxy-64545848b5-ffqsz\" (UID: \"4e810e5c-e197-4b47-b906-95e8b88f5586\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"
Oct 29 21:50:23.254240 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.254208    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e810e5c-e197-4b47-b906-95e8b88f5586-tmp\") pod \"kube-auth-proxy-64545848b5-ffqsz\" (UID: \"4e810e5c-e197-4b47-b906-95e8b88f5586\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"
Oct 29 21:50:23.254374 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.254360    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e810e5c-e197-4b47-b906-95e8b88f5586-tls-certs\") pod \"kube-auth-proxy-64545848b5-ffqsz\" (UID: \"4e810e5c-e197-4b47-b906-95e8b88f5586\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"
Oct 29 21:50:23.278875 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.278848    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hqq4\" (UniqueName: \"kubernetes.io/projected/4e810e5c-e197-4b47-b906-95e8b88f5586-kube-api-access-9hqq4\") pod \"kube-auth-proxy-64545848b5-ffqsz\" (UID: \"4e810e5c-e197-4b47-b906-95e8b88f5586\") " pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"
Oct 29 21:50:23.319830 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.319796    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"]
Oct 29 21:50:23.323730 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.323705    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.341188 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.341162    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-cv2hc\""
Oct 29 21:50:23.350702 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.350667    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"]
Oct 29 21:50:23.352996 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.352970    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.353132 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.353018    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.353132 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.353047    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98p9d\" (UniqueName: \"kubernetes.io/projected/16d41fba-a15d-4b5c-b42f-54457db55a5a-kube-api-access-98p9d\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.353220 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.353130    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.353220 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.353198    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.353295 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.353282    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.353328 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.353308    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d41fba-a15d-4b5c-b42f-54457db55a5a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.353404 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.353389    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d41fba-a15d-4b5c-b42f-54457db55a5a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.353449 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.353414    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d41fba-a15d-4b5c-b42f-54457db55a5a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.450576 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.450489    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"
Oct 29 21:50:23.454760 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.454730    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.454898 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.454779    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98p9d\" (UniqueName: \"kubernetes.io/projected/16d41fba-a15d-4b5c-b42f-54457db55a5a-kube-api-access-98p9d\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.454898 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.454806    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.454898 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.454833    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.455066 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.454910    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.455066 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.454934    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d41fba-a15d-4b5c-b42f-54457db55a5a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.455066 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.454985    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d41fba-a15d-4b5c-b42f-54457db55a5a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.455066 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.455011    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d41fba-a15d-4b5c-b42f-54457db55a5a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.455276 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.455064    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.455276 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.455255    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.455574 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.455531    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.455702 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.455569    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.455850 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.455820    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.456166 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.456145    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16d41fba-a15d-4b5c-b42f-54457db55a5a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.457994 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.457967    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16d41fba-a15d-4b5c-b42f-54457db55a5a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.458123 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.458104    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16d41fba-a15d-4b5c-b42f-54457db55a5a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.478174 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.478148    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98p9d\" (UniqueName: \"kubernetes.io/projected/16d41fba-a15d-4b5c-b42f-54457db55a5a-kube-api-access-98p9d\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.478354 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.478295    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16d41fba-a15d-4b5c-b42f-54457db55a5a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5j9s7t\" (UID: \"16d41fba-a15d-4b5c-b42f-54457db55a5a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.640113 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.640081    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:23.756379 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.756287    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64545848b5-ffqsz"]
Oct 29 21:50:23.760045 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:50:23.760008    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e810e5c_e197_4b47_b906_95e8b88f5586.slice/crio-ac25b2f00a3cbc34ff9eb821758449b9a31e3e6a0dfcbb1f0b21847ffe8657f0 WatchSource:0}: Error finding container ac25b2f00a3cbc34ff9eb821758449b9a31e3e6a0dfcbb1f0b21847ffe8657f0: Status 404 returned error can't find the container with id ac25b2f00a3cbc34ff9eb821758449b9a31e3e6a0dfcbb1f0b21847ffe8657f0
Oct 29 21:50:23.953190 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.953161    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"]
Oct 29 21:50:23.955649 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:50:23.955610    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d41fba_a15d_4b5c_b42f_54457db55a5a.slice/crio-384119bf6d525a7698423ce76be391d50709c96addc8faef72fb1e344094e5a6 WatchSource:0}: Error finding container 384119bf6d525a7698423ce76be391d50709c96addc8faef72fb1e344094e5a6: Status 404 returned error can't find the container with id 384119bf6d525a7698423ce76be391d50709c96addc8faef72fb1e344094e5a6
Oct 29 21:50:23.957973 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.957937    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:50:23.958058 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.958012    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:50:23.958058 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:23.958041    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:50:24.770713 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:24.770659    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t" event={"ID":"16d41fba-a15d-4b5c-b42f-54457db55a5a","Type":"ContainerStarted","Data":"58f7a8acf8dc414847069336cb3dc03448affdbb1b5ff9fd6f6ce87c5e475dd2"}
Oct 29 21:50:24.771249 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:24.770734    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t" event={"ID":"16d41fba-a15d-4b5c-b42f-54457db55a5a","Type":"ContainerStarted","Data":"384119bf6d525a7698423ce76be391d50709c96addc8faef72fb1e344094e5a6"}
Oct 29 21:50:24.772242 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:24.772215    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz" event={"ID":"4e810e5c-e197-4b47-b906-95e8b88f5586","Type":"ContainerStarted","Data":"ac25b2f00a3cbc34ff9eb821758449b9a31e3e6a0dfcbb1f0b21847ffe8657f0"}
Oct 29 21:50:24.840502 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:24.840434    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t" podStartSLOduration=1.840409757 podStartE2EDuration="1.840409757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:24.828191787 +0000 UTC m=+197.328234093" watchObservedRunningTime="2026-10-29 21:50:24.840409757 +0000 UTC m=+197.340452061"
Oct 29 21:50:25.640318 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:25.640286    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:25.645564 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:25.645536    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:25.778105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:25.778063    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:25.779657 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:25.779607    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5j9s7t"
Oct 29 21:50:26.790077 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:26.790036    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz" event={"ID":"4e810e5c-e197-4b47-b906-95e8b88f5586","Type":"ContainerStarted","Data":"27d3c3b209937d7ab693aa9309aa7484a7e1e7b403033738f55d033328ab07ba"}
Oct 29 21:50:26.833244 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:26.833182    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64545848b5-ffqsz" podStartSLOduration=1.520773787 podStartE2EDuration="3.833168359s" totalImagesPullingTime="2.312394572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:26.831738101 +0000 UTC m=+199.331780400" watchObservedRunningTime="2026-10-29 21:50:26.833168359 +0000 UTC m=+199.333210658"
Oct 29 21:50:28.413881 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:50:28.413849    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:50:29.294782 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:29.294744    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/developer-portal-controller-66d8dbb8c8-b2ll6"]
Oct 29 21:50:29.299653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:29.299616    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-b2ll6"
Oct 29 21:50:29.319306 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:29.319274    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/developer-portal-controller-66d8dbb8c8-b2ll6"]
Oct 29 21:50:29.326309 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:29.326285    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"developer-portal-controller-manager-dockercfg-t6bsh\""
Oct 29 21:50:29.420811 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:29.420776    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8k9r\" (UniqueName: \"kubernetes.io/projected/ba1d0015-3c5e-450d-a435-d33ab885f265-kube-api-access-q8k9r\") pod \"developer-portal-controller-66d8dbb8c8-b2ll6\" (UID: \"ba1d0015-3c5e-450d-a435-d33ab885f265\") " pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-b2ll6"
Oct 29 21:50:29.521583 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:29.521549    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8k9r\" (UniqueName: \"kubernetes.io/projected/ba1d0015-3c5e-450d-a435-d33ab885f265-kube-api-access-q8k9r\") pod \"developer-portal-controller-66d8dbb8c8-b2ll6\" (UID: \"ba1d0015-3c5e-450d-a435-d33ab885f265\") " pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-b2ll6"
Oct 29 21:50:29.545342 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:29.545265    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8k9r\" (UniqueName: \"kubernetes.io/projected/ba1d0015-3c5e-450d-a435-d33ab885f265-kube-api-access-q8k9r\") pod \"developer-portal-controller-66d8dbb8c8-b2ll6\" (UID: \"ba1d0015-3c5e-450d-a435-d33ab885f265\") " pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-b2ll6"
Oct 29 21:50:29.613309 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:29.613278    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-b2ll6"
Oct 29 21:50:29.912848 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:29.912646    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/developer-portal-controller-66d8dbb8c8-b2ll6"]
Oct 29 21:50:29.915608 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:50:29.915575    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1d0015_3c5e_450d_a435_d33ab885f265.slice/crio-c5175d4702c8c7a0e4881dda36fde01fb486b3fc261f98dac05550570661a18f WatchSource:0}: Error finding container c5175d4702c8c7a0e4881dda36fde01fb486b3fc261f98dac05550570661a18f: Status 404 returned error can't find the container with id c5175d4702c8c7a0e4881dda36fde01fb486b3fc261f98dac05550570661a18f
Oct 29 21:50:30.812093 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:30.812053    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-b2ll6" event={"ID":"ba1d0015-3c5e-450d-a435-d33ab885f265","Type":"ContainerStarted","Data":"c5175d4702c8c7a0e4881dda36fde01fb486b3fc261f98dac05550570661a18f"}
Oct 29 21:50:31.818390 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:31.818355    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-b2ll6" event={"ID":"ba1d0015-3c5e-450d-a435-d33ab885f265","Type":"ContainerStarted","Data":"ed80ad59e28be039544fa4e03cbc72f8d12dbb6e056b5ff34cd04cd86cc8a2eb"}
Oct 29 21:50:31.818946 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:31.818472    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-b2ll6"
Oct 29 21:50:31.859159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:31.859104    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-b2ll6" podStartSLOduration=1.033950618 podStartE2EDuration="2.859088834s" totalImagesPullingTime="1.825138216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:31.858577294 +0000 UTC m=+204.358619597" watchObservedRunningTime="2026-10-29 21:50:31.859088834 +0000 UTC m=+204.359131133"
Oct 29 21:50:32.323382 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.323348    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"]
Oct 29 21:50:32.327366 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.327344    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.344150 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.344128    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 29 21:50:32.355906 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.355857    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 29 21:50:32.355906 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.355884    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-rhntg\""
Oct 29 21:50:32.380544 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.380508    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 29 21:50:32.381747 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.381726    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"]
Oct 29 21:50:32.449738 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.449706    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/522ce2f8-9d09-478b-8575-b2dabf74a61e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9cw8s\" (UID: \"522ce2f8-9d09-478b-8575-b2dabf74a61e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.449927 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.449757    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7x4t\" (UniqueName: \"kubernetes.io/projected/522ce2f8-9d09-478b-8575-b2dabf74a61e-kube-api-access-x7x4t\") pod \"kserve-module-controller-manager-7566cf5fc8-9cw8s\" (UID: \"522ce2f8-9d09-478b-8575-b2dabf74a61e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.449927 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.449780    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/522ce2f8-9d09-478b-8575-b2dabf74a61e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9cw8s\" (UID: \"522ce2f8-9d09-478b-8575-b2dabf74a61e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.450020 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.449928    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/522ce2f8-9d09-478b-8575-b2dabf74a61e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9cw8s\" (UID: \"522ce2f8-9d09-478b-8575-b2dabf74a61e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.550745 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.550709    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/522ce2f8-9d09-478b-8575-b2dabf74a61e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9cw8s\" (UID: \"522ce2f8-9d09-478b-8575-b2dabf74a61e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.550961 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.550759    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7x4t\" (UniqueName: \"kubernetes.io/projected/522ce2f8-9d09-478b-8575-b2dabf74a61e-kube-api-access-x7x4t\") pod \"kserve-module-controller-manager-7566cf5fc8-9cw8s\" (UID: \"522ce2f8-9d09-478b-8575-b2dabf74a61e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.550961 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.550782    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/522ce2f8-9d09-478b-8575-b2dabf74a61e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9cw8s\" (UID: \"522ce2f8-9d09-478b-8575-b2dabf74a61e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.550961 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.550829    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/522ce2f8-9d09-478b-8575-b2dabf74a61e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9cw8s\" (UID: \"522ce2f8-9d09-478b-8575-b2dabf74a61e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.551359 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.551334    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/522ce2f8-9d09-478b-8575-b2dabf74a61e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9cw8s\" (UID: \"522ce2f8-9d09-478b-8575-b2dabf74a61e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.551450 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.551343    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/522ce2f8-9d09-478b-8575-b2dabf74a61e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9cw8s\" (UID: \"522ce2f8-9d09-478b-8575-b2dabf74a61e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.551749 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.551726    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/522ce2f8-9d09-478b-8575-b2dabf74a61e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9cw8s\" (UID: \"522ce2f8-9d09-478b-8575-b2dabf74a61e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.574808 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.574740    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7x4t\" (UniqueName: \"kubernetes.io/projected/522ce2f8-9d09-478b-8575-b2dabf74a61e-kube-api-access-x7x4t\") pod \"kserve-module-controller-manager-7566cf5fc8-9cw8s\" (UID: \"522ce2f8-9d09-478b-8575-b2dabf74a61e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.641013 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.640981    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:32.758235 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.758205    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8"
Oct 29 21:50:32.960163 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:32.960078    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"]
Oct 29 21:50:32.963321 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:50:32.963299    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522ce2f8_9d09_478b_8575_b2dabf74a61e.slice/crio-ed6adfbfeb4a3874acc7ffdcaca5132fdf6a92b59e729457934b57b7d63e1192 WatchSource:0}: Error finding container ed6adfbfeb4a3874acc7ffdcaca5132fdf6a92b59e729457934b57b7d63e1192: Status 404 returned error can't find the container with id ed6adfbfeb4a3874acc7ffdcaca5132fdf6a92b59e729457934b57b7d63e1192
Oct 29 21:50:33.847501 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:33.847470    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s" event={"ID":"522ce2f8-9d09-478b-8575-b2dabf74a61e","Type":"ContainerStarted","Data":"ed6adfbfeb4a3874acc7ffdcaca5132fdf6a92b59e729457934b57b7d63e1192"}
Oct 29 21:50:34.853608 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:34.853574    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s" event={"ID":"522ce2f8-9d09-478b-8575-b2dabf74a61e","Type":"ContainerStarted","Data":"cabf5037320fc0fc0d32fa8b9241877ba8386fd3516c2dff9d337f65cf1db5c4"}
Oct 29 21:50:34.854020 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:34.853687    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:34.909709 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:34.909652    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s" podStartSLOduration=1.407933182 podStartE2EDuration="2.909621966s" totalImagesPullingTime="1.501688784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:34.909082537 +0000 UTC m=+207.409124872" watchObservedRunningTime="2026-10-29 21:50:34.909621966 +0000 UTC m=+207.409664258"
Oct 29 21:50:42.844258 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:42.844227    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/developer-portal-controller-66d8dbb8c8-b2ll6"
Oct 29 21:50:44.022211 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.022175    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:50:44.025217 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.025194    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"
Oct 29 21:50:44.040510 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.040473    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 29 21:50:44.052262 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.052235    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8846b\""
Oct 29 21:50:44.053908 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.053888    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:50:44.122317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.122287    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:50:44.163770 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.163740    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/72d022e0-2c45-448f-8bfb-77d5d75d6f8f-config-file\") pod \"limitador-limitador-f7d6b5c7c-7sl7d\" (UID: \"72d022e0-2c45-448f-8bfb-77d5d75d6f8f\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"
Oct 29 21:50:44.163928 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.163798    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt9rr\" (UniqueName: \"kubernetes.io/projected/72d022e0-2c45-448f-8bfb-77d5d75d6f8f-kube-api-access-zt9rr\") pod \"limitador-limitador-f7d6b5c7c-7sl7d\" (UID: \"72d022e0-2c45-448f-8bfb-77d5d75d6f8f\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"
Oct 29 21:50:44.264685 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.264643    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt9rr\" (UniqueName: \"kubernetes.io/projected/72d022e0-2c45-448f-8bfb-77d5d75d6f8f-kube-api-access-zt9rr\") pod \"limitador-limitador-f7d6b5c7c-7sl7d\" (UID: \"72d022e0-2c45-448f-8bfb-77d5d75d6f8f\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"
Oct 29 21:50:44.264906 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.264791    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/72d022e0-2c45-448f-8bfb-77d5d75d6f8f-config-file\") pod \"limitador-limitador-f7d6b5c7c-7sl7d\" (UID: \"72d022e0-2c45-448f-8bfb-77d5d75d6f8f\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"
Oct 29 21:50:44.265399 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.265380    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/72d022e0-2c45-448f-8bfb-77d5d75d6f8f-config-file\") pod \"limitador-limitador-f7d6b5c7c-7sl7d\" (UID: \"72d022e0-2c45-448f-8bfb-77d5d75d6f8f\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"
Oct 29 21:50:44.288285 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.288226    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt9rr\" (UniqueName: \"kubernetes.io/projected/72d022e0-2c45-448f-8bfb-77d5d75d6f8f-kube-api-access-zt9rr\") pod \"limitador-limitador-f7d6b5c7c-7sl7d\" (UID: \"72d022e0-2c45-448f-8bfb-77d5d75d6f8f\") " pod="kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"
Oct 29 21:50:44.337056 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.337027    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"
Oct 29 21:50:44.640694 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.640666    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:50:44.643389 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:50:44.643358    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d022e0_2c45_448f_8bfb_77d5d75d6f8f.slice/crio-78dc20e603ef066d779302497cd35cf821ce26c0ad8568c85a553cf699496961 WatchSource:0}: Error finding container 78dc20e603ef066d779302497cd35cf821ce26c0ad8568c85a553cf699496961: Status 404 returned error can't find the container with id 78dc20e603ef066d779302497cd35cf821ce26c0ad8568c85a553cf699496961
Oct 29 21:50:44.906602 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:44.906521    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d" event={"ID":"72d022e0-2c45-448f-8bfb-77d5d75d6f8f","Type":"ContainerStarted","Data":"78dc20e603ef066d779302497cd35cf821ce26c0ad8568c85a553cf699496961"}
Oct 29 21:50:45.404482 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:45.404452    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-dc8775675-wnqnl"]
Oct 29 21:50:45.408415 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:45.408391    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc8775675-wnqnl"
Oct 29 21:50:45.416742 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:45.416712    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dc8775675-wnqnl"]
Oct 29 21:50:45.426413 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:45.426378    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-h7r8w\""
Oct 29 21:50:45.476428 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:45.476396    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82mp4\" (UniqueName: \"kubernetes.io/projected/7428044b-b88e-4e73-84f8-d367a812703a-kube-api-access-82mp4\") pod \"authorino-dc8775675-wnqnl\" (UID: \"7428044b-b88e-4e73-84f8-d367a812703a\") " pod="kuadrant-system/authorino-dc8775675-wnqnl"
Oct 29 21:50:45.577148 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:45.577118    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82mp4\" (UniqueName: \"kubernetes.io/projected/7428044b-b88e-4e73-84f8-d367a812703a-kube-api-access-82mp4\") pod \"authorino-dc8775675-wnqnl\" (UID: \"7428044b-b88e-4e73-84f8-d367a812703a\") " pod="kuadrant-system/authorino-dc8775675-wnqnl"
Oct 29 21:50:45.600114 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:45.600085    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82mp4\" (UniqueName: \"kubernetes.io/projected/7428044b-b88e-4e73-84f8-d367a812703a-kube-api-access-82mp4\") pod \"authorino-dc8775675-wnqnl\" (UID: \"7428044b-b88e-4e73-84f8-d367a812703a\") " pod="kuadrant-system/authorino-dc8775675-wnqnl"
Oct 29 21:50:45.723543 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:45.723455    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc8775675-wnqnl"
Oct 29 21:50:45.914772 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:45.914739    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d" event={"ID":"72d022e0-2c45-448f-8bfb-77d5d75d6f8f","Type":"ContainerStarted","Data":"3b6eeda6fda66db179c7262088d3b7bec04b9a0ca24061eb5e7aafa49cb0286c"}
Oct 29 21:50:45.914958 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:45.914853    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"
Oct 29 21:50:45.956550 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:45.956502    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d" podStartSLOduration=2.092454224 podStartE2EDuration="2.956486765s" totalImagesPullingTime="864.032541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:45.95573439 +0000 UTC m=+218.455776686" watchObservedRunningTime="2026-10-29 21:50:45.956486765 +0000 UTC m=+218.456529077"
Oct 29 21:50:46.022035 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:46.021957    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dc8775675-wnqnl"]
Oct 29 21:50:46.025031 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:50:46.024998    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7428044b_b88e_4e73_84f8_d367a812703a.slice/crio-f830f7d11e1cc1c599014866c6d92956dae5f1750ab5f9bb9ef93b97c20f5165 WatchSource:0}: Error finding container f830f7d11e1cc1c599014866c6d92956dae5f1750ab5f9bb9ef93b97c20f5165: Status 404 returned error can't find the container with id f830f7d11e1cc1c599014866c6d92956dae5f1750ab5f9bb9ef93b97c20f5165
Oct 29 21:50:46.923086 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:46.923044    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc8775675-wnqnl" event={"ID":"7428044b-b88e-4e73-84f8-d367a812703a","Type":"ContainerStarted","Data":"f830f7d11e1cc1c599014866c6d92956dae5f1750ab5f9bb9ef93b97c20f5165"}
Oct 29 21:50:48.935391 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:48.935351    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc8775675-wnqnl" event={"ID":"7428044b-b88e-4e73-84f8-d367a812703a","Type":"ContainerStarted","Data":"575362ae24ca55d04b09b16876041b01ab90e0599c0f38791bbdfa52b73b035b"}
Oct 29 21:50:48.975822 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:48.975763    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-dc8775675-wnqnl" podStartSLOduration=1.662349411 podStartE2EDuration="3.975749203s" totalImagesPullingTime="2.313399792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:48.974317767 +0000 UTC m=+221.474360066" watchObservedRunningTime="2026-10-29 21:50:48.975749203 +0000 UTC m=+221.475791502"
Oct 29 21:50:55.860240 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:55.860208    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9cw8s"
Oct 29 21:50:56.924501 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:50:56.924470    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"
Oct 29 21:50:58.411534 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:50:58.411489    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:51:28.414822 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:51:28.414777    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:51:57.108823 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:51:57.108784    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:51:58.429331 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:51:58.429293    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:52:05.436543 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.436507    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7597569d74-c9hhs"]
Oct 29 21:52:05.441211 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.441183    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597569d74-c9hhs"
Oct 29 21:52:05.465985 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.465950    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7597569d74-c9hhs"]
Oct 29 21:52:05.568876 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.568840    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sklgr\" (UniqueName: \"kubernetes.io/projected/aed10091-095d-4fc6-84d0-8885cf64235d-kube-api-access-sklgr\") pod \"authorino-7597569d74-c9hhs\" (UID: \"aed10091-095d-4fc6-84d0-8885cf64235d\") " pod="kuadrant-system/authorino-7597569d74-c9hhs"
Oct 29 21:52:05.609674 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.609621    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7597569d74-c9hhs"]
Oct 29 21:52:05.609963 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:52:05.609937    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-sklgr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7597569d74-c9hhs" podUID="aed10091-095d-4fc6-84d0-8885cf64235d"
Oct 29 21:52:05.658479 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.658445    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65db9f99d7-q88q2"]
Oct 29 21:52:05.662798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.662772    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65db9f99d7-q88q2"
Oct 29 21:52:05.669742 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.669714    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sklgr\" (UniqueName: \"kubernetes.io/projected/aed10091-095d-4fc6-84d0-8885cf64235d-kube-api-access-sklgr\") pod \"authorino-7597569d74-c9hhs\" (UID: \"aed10091-095d-4fc6-84d0-8885cf64235d\") " pod="kuadrant-system/authorino-7597569d74-c9hhs"
Oct 29 21:52:05.682815 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.682786    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65db9f99d7-q88q2"]
Oct 29 21:52:05.693765 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.693697    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sklgr\" (UniqueName: \"kubernetes.io/projected/aed10091-095d-4fc6-84d0-8885cf64235d-kube-api-access-sklgr\") pod \"authorino-7597569d74-c9hhs\" (UID: \"aed10091-095d-4fc6-84d0-8885cf64235d\") " pod="kuadrant-system/authorino-7597569d74-c9hhs"
Oct 29 21:52:05.764318 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.764291    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 29 21:52:05.770707 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.770672    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a22958c-05b4-4f57-98fd-c953e0668299-tls-cert\") pod \"authorino-65db9f99d7-q88q2\" (UID: \"1a22958c-05b4-4f57-98fd-c953e0668299\") " pod="kuadrant-system/authorino-65db9f99d7-q88q2"
Oct 29 21:52:05.770850 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.770721    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6nt\" (UniqueName: \"kubernetes.io/projected/1a22958c-05b4-4f57-98fd-c953e0668299-kube-api-access-fq6nt\") pod \"authorino-65db9f99d7-q88q2\" (UID: \"1a22958c-05b4-4f57-98fd-c953e0668299\") " pod="kuadrant-system/authorino-65db9f99d7-q88q2"
Oct 29 21:52:05.803392 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.803350    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65db9f99d7-q88q2"]
Oct 29 21:52:05.803710 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:52:05.803684    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fq6nt tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-65db9f99d7-q88q2" podUID="1a22958c-05b4-4f57-98fd-c953e0668299"
Oct 29 21:52:05.872019 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.871981    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6nt\" (UniqueName: \"kubernetes.io/projected/1a22958c-05b4-4f57-98fd-c953e0668299-kube-api-access-fq6nt\") pod \"authorino-65db9f99d7-q88q2\" (UID: \"1a22958c-05b4-4f57-98fd-c953e0668299\") " pod="kuadrant-system/authorino-65db9f99d7-q88q2"
Oct 29 21:52:05.872225 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.872121    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a22958c-05b4-4f57-98fd-c953e0668299-tls-cert\") pod \"authorino-65db9f99d7-q88q2\" (UID: \"1a22958c-05b4-4f57-98fd-c953e0668299\") " pod="kuadrant-system/authorino-65db9f99d7-q88q2"
Oct 29 21:52:05.875011 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.874986    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a22958c-05b4-4f57-98fd-c953e0668299-tls-cert\") pod \"authorino-65db9f99d7-q88q2\" (UID: \"1a22958c-05b4-4f57-98fd-c953e0668299\") " pod="kuadrant-system/authorino-65db9f99d7-q88q2"
Oct 29 21:52:05.897445 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:05.897407    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6nt\" (UniqueName: \"kubernetes.io/projected/1a22958c-05b4-4f57-98fd-c953e0668299-kube-api-access-fq6nt\") pod \"authorino-65db9f99d7-q88q2\" (UID: \"1a22958c-05b4-4f57-98fd-c953e0668299\") " pod="kuadrant-system/authorino-65db9f99d7-q88q2"
Oct 29 21:52:06.355382 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.355351    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65db9f99d7-q88q2"
Oct 29 21:52:06.355382 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.355374    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597569d74-c9hhs"
Oct 29 21:52:06.364819 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.364793    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65db9f99d7-q88q2"
Oct 29 21:52:06.368918 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.368894    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597569d74-c9hhs"
Oct 29 21:52:06.438768 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.438734    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f556s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 21:52:06.439163 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.438782    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" podUID="3c28b204-2e05-473f-b027-13d086286e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:52:06.478311 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.478281    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed10091-095d-4fc6-84d0-8885cf64235d-kube-api-access-sklgr\" (UniqueName: \"kubernetes.io/projected/aed10091-095d-4fc6-84d0-8885cf64235d-kube-api-access-sklgr\") pod \"aed10091-095d-4fc6-84d0-8885cf64235d\" (UID: \"aed10091-095d-4fc6-84d0-8885cf64235d\") "
Oct 29 21:52:06.478311 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.478315    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a22958c-05b4-4f57-98fd-c953e0668299-kube-api-access-fq6nt\" (UniqueName: \"kubernetes.io/projected/1a22958c-05b4-4f57-98fd-c953e0668299-kube-api-access-fq6nt\") pod \"1a22958c-05b4-4f57-98fd-c953e0668299\" (UID: \"1a22958c-05b4-4f57-98fd-c953e0668299\") "
Oct 29 21:52:06.478585 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.478463    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a22958c-05b4-4f57-98fd-c953e0668299-tls-cert\" (UniqueName: \"kubernetes.io/secret/1a22958c-05b4-4f57-98fd-c953e0668299-tls-cert\") pod \"1a22958c-05b4-4f57-98fd-c953e0668299\" (UID: \"1a22958c-05b4-4f57-98fd-c953e0668299\") "
Oct 29 21:52:06.480671 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.480616    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a22958c-05b4-4f57-98fd-c953e0668299-tls-cert" pod "1a22958c-05b4-4f57-98fd-c953e0668299" (UID: "1a22958c-05b4-4f57-98fd-c953e0668299"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:52:06.480815 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.480765    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a22958c-05b4-4f57-98fd-c953e0668299-kube-api-access-fq6nt" pod "1a22958c-05b4-4f57-98fd-c953e0668299" (UID: "1a22958c-05b4-4f57-98fd-c953e0668299"). InnerVolumeSpecName "kube-api-access-fq6nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:52:06.481012 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.480993    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed10091-095d-4fc6-84d0-8885cf64235d-kube-api-access-sklgr" pod "aed10091-095d-4fc6-84d0-8885cf64235d" (UID: "aed10091-095d-4fc6-84d0-8885cf64235d"). InnerVolumeSpecName "kube-api-access-sklgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:52:06.580383 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.580340    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sklgr\" (UniqueName: \"kubernetes.io/projected/aed10091-095d-4fc6-84d0-8885cf64235d-kube-api-access-sklgr\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:52:06.580383 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.580385    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq6nt\" (UniqueName: \"kubernetes.io/projected/1a22958c-05b4-4f57-98fd-c953e0668299-kube-api-access-fq6nt\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:52:06.580782 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.580402    7082 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a22958c-05b4-4f57-98fd-c953e0668299-tls-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:52:06.715737 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.715659    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8687494d7b-rpm8k"]
Oct 29 21:52:06.719159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.719140    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8687494d7b-rpm8k"
Oct 29 21:52:06.739344 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.739319    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8687494d7b-rpm8k"]
Oct 29 21:52:06.882886 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.882853    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqt2h\" (UniqueName: \"kubernetes.io/projected/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-kube-api-access-rqt2h\") pod \"authorino-8687494d7b-rpm8k\" (UID: \"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2\") " pod="kuadrant-system/authorino-8687494d7b-rpm8k"
Oct 29 21:52:06.883088 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.882901    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-tls-cert\") pod \"authorino-8687494d7b-rpm8k\" (UID: \"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2\") " pod="kuadrant-system/authorino-8687494d7b-rpm8k"
Oct 29 21:52:06.983870 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.983785    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqt2h\" (UniqueName: \"kubernetes.io/projected/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-kube-api-access-rqt2h\") pod \"authorino-8687494d7b-rpm8k\" (UID: \"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2\") " pod="kuadrant-system/authorino-8687494d7b-rpm8k"
Oct 29 21:52:06.983870 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.983831    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-tls-cert\") pod \"authorino-8687494d7b-rpm8k\" (UID: \"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2\") " pod="kuadrant-system/authorino-8687494d7b-rpm8k"
Oct 29 21:52:06.986589 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:06.986566    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-tls-cert\") pod \"authorino-8687494d7b-rpm8k\" (UID: \"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2\") " pod="kuadrant-system/authorino-8687494d7b-rpm8k"
Oct 29 21:52:07.007335 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.007299    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqt2h\" (UniqueName: \"kubernetes.io/projected/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-kube-api-access-rqt2h\") pod \"authorino-8687494d7b-rpm8k\" (UID: \"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2\") " pod="kuadrant-system/authorino-8687494d7b-rpm8k"
Oct 29 21:52:07.030776 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.030749    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8687494d7b-rpm8k"
Oct 29 21:52:07.347744 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.347709    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8687494d7b-rpm8k"]
Oct 29 21:52:07.350720 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:52:07.350692    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f1f5993_efcc_42f8_9b47_0e0e4e9f98d2.slice/crio-2241dd442a0c9703b5437d8e558bf1fa02bf86993ebd6f3900bfcd08136cd668 WatchSource:0}: Error finding container 2241dd442a0c9703b5437d8e558bf1fa02bf86993ebd6f3900bfcd08136cd668: Status 404 returned error can't find the container with id 2241dd442a0c9703b5437d8e558bf1fa02bf86993ebd6f3900bfcd08136cd668
Oct 29 21:52:07.362488 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.362406    7082 generic.go:356] "Generic (PLEG): container finished" podID="3c28b204-2e05-473f-b027-13d086286e51" containerID="f1ff411dd64f982b0603fb68bd3230ba10852f736d1c301d91ebe48d946a438d" exitCode=0
Oct 29 21:52:07.362656 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.362486    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" event={"ID":"3c28b204-2e05-473f-b027-13d086286e51","Type":"ContainerDied","Data":"f1ff411dd64f982b0603fb68bd3230ba10852f736d1c301d91ebe48d946a438d"}
Oct 29 21:52:07.362964 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.362944    7082 scope.go:122] "RemoveContainer" containerID="f1ff411dd64f982b0603fb68bd3230ba10852f736d1c301d91ebe48d946a438d"
Oct 29 21:52:07.363873 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.363853    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7597569d74-c9hhs"
Oct 29 21:52:07.363975 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.363872    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8687494d7b-rpm8k" event={"ID":"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2","Type":"ContainerStarted","Data":"2241dd442a0c9703b5437d8e558bf1fa02bf86993ebd6f3900bfcd08136cd668"}
Oct 29 21:52:07.363975 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.363945    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65db9f99d7-q88q2"
Oct 29 21:52:07.364195 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.364156    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:52:07.364316 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.364255    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:52:07.364375 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.364309    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:52:07.477783 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.477693    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7597569d74-c9hhs"]
Oct 29 21:52:07.479142 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.478371    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7597569d74-c9hhs"]
Oct 29 21:52:07.538062 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.538033    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65db9f99d7-q88q2"]
Oct 29 21:52:07.543272 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.543173    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65db9f99d7-q88q2"]
Oct 29 21:52:07.635686 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.635576    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a22958c-05b4-4f57-98fd-c953e0668299" path="/var/lib/kubelet/pods/1a22958c-05b4-4f57-98fd-c953e0668299/volumes"
Oct 29 21:52:07.636003 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:07.635985    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed10091-095d-4fc6-84d0-8885cf64235d" path="/var/lib/kubelet/pods/aed10091-095d-4fc6-84d0-8885cf64235d/volumes"
Oct 29 21:52:08.373120 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:08.373086    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" event={"ID":"3c28b204-2e05-473f-b027-13d086286e51","Type":"ContainerStarted","Data":"3ca400ba5a58244757b42cab9c51199345ab4cb6f5b471981cdfbe6810dff964"}
Oct 29 21:52:08.375054 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:08.375024    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8687494d7b-rpm8k" event={"ID":"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2","Type":"ContainerStarted","Data":"299650874af8335f64c7d7a11fbee74904e7652291e6619da6e95b047055aa74"}
Oct 29 21:52:08.476668 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:08.475601    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-8687494d7b-rpm8k" podStartSLOduration=1.996425495 podStartE2EDuration="2.475584017s" totalImagesPullingTime="479.158522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:08.47507131 +0000 UTC m=+300.975113610" watchObservedRunningTime="2026-10-29 21:52:08.475584017 +0000 UTC m=+300.975626318"
Oct 29 21:52:08.523656 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:08.520124    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-dc8775675-wnqnl"]
Oct 29 21:52:08.523656 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:08.520341    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-dc8775675-wnqnl" podUID="7428044b-b88e-4e73-84f8-d367a812703a" containerName="authorino" containerID="cri-o://575362ae24ca55d04b09b16876041b01ab90e0599c0f38791bbdfa52b73b035b" gracePeriod=30
Oct 29 21:52:08.850367 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:08.850344    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc8775675-wnqnl"
Oct 29 21:52:09.027183 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.027082    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7428044b-b88e-4e73-84f8-d367a812703a-kube-api-access-82mp4\" (UniqueName: \"kubernetes.io/projected/7428044b-b88e-4e73-84f8-d367a812703a-kube-api-access-82mp4\") pod \"7428044b-b88e-4e73-84f8-d367a812703a\" (UID: \"7428044b-b88e-4e73-84f8-d367a812703a\") "
Oct 29 21:52:09.030064 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.030039    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7428044b-b88e-4e73-84f8-d367a812703a-kube-api-access-82mp4" pod "7428044b-b88e-4e73-84f8-d367a812703a" (UID: "7428044b-b88e-4e73-84f8-d367a812703a"). InnerVolumeSpecName "kube-api-access-82mp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:52:09.128371 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.128326    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82mp4\" (UniqueName: \"kubernetes.io/projected/7428044b-b88e-4e73-84f8-d367a812703a-kube-api-access-82mp4\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:52:09.183068 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.183029    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:52:09.183246 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.183092    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f556s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Oct 29 21:52:09.183246 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.183170    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" podUID="3c28b204-2e05-473f-b027-13d086286e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Oct 29 21:52:09.381894 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.381859    7082 generic.go:356] "Generic (PLEG): container finished" podID="7428044b-b88e-4e73-84f8-d367a812703a" containerID="575362ae24ca55d04b09b16876041b01ab90e0599c0f38791bbdfa52b73b035b" exitCode=0
Oct 29 21:52:09.382082 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.381915    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dc8775675-wnqnl"
Oct 29 21:52:09.382082 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.381955    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc8775675-wnqnl" event={"ID":"7428044b-b88e-4e73-84f8-d367a812703a","Type":"ContainerDied","Data":"575362ae24ca55d04b09b16876041b01ab90e0599c0f38791bbdfa52b73b035b"}
Oct 29 21:52:09.382082 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.382007    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dc8775675-wnqnl" event={"ID":"7428044b-b88e-4e73-84f8-d367a812703a","Type":"ContainerDied","Data":"f830f7d11e1cc1c599014866c6d92956dae5f1750ab5f9bb9ef93b97c20f5165"}
Oct 29 21:52:09.382082 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.382029    7082 scope.go:122] "RemoveContainer" containerID="575362ae24ca55d04b09b16876041b01ab90e0599c0f38791bbdfa52b73b035b"
Oct 29 21:52:09.421666 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.421648    7082 scope.go:122] "RemoveContainer" containerID="575362ae24ca55d04b09b16876041b01ab90e0599c0f38791bbdfa52b73b035b"
Oct 29 21:52:09.422024 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:52:09.422003    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575362ae24ca55d04b09b16876041b01ab90e0599c0f38791bbdfa52b73b035b\": container with ID starting with 575362ae24ca55d04b09b16876041b01ab90e0599c0f38791bbdfa52b73b035b not found: ID does not exist" containerID="575362ae24ca55d04b09b16876041b01ab90e0599c0f38791bbdfa52b73b035b"
Oct 29 21:52:09.422078 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.422035    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575362ae24ca55d04b09b16876041b01ab90e0599c0f38791bbdfa52b73b035b"} err="failed to get container status \"575362ae24ca55d04b09b16876041b01ab90e0599c0f38791bbdfa52b73b035b\": rpc error: code = NotFound desc = could not find container \"575362ae24ca55d04b09b16876041b01ab90e0599c0f38791bbdfa52b73b035b\": container with ID starting with 575362ae24ca55d04b09b16876041b01ab90e0599c0f38791bbdfa52b73b035b not found: ID does not exist"
Oct 29 21:52:09.452637 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.452592    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-dc8775675-wnqnl"]
Oct 29 21:52:09.457106 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.457077    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-dc8775675-wnqnl"]
Oct 29 21:52:09.630359 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:09.630327    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7428044b-b88e-4e73-84f8-d367a812703a" path="/var/lib/kubelet/pods/7428044b-b88e-4e73-84f8-d367a812703a/volumes"
Oct 29 21:52:10.182243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:10.182210    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f556s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Oct 29 21:52:10.182419 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:10.182276    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" podUID="3c28b204-2e05-473f-b027-13d086286e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Oct 29 21:52:11.182095 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:11.182065    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f556s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Oct 29 21:52:11.182482 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:11.182119    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" podUID="3c28b204-2e05-473f-b027-13d086286e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Oct 29 21:52:12.182468 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:12.182426    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f556s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Oct 29 21:52:12.182895 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:12.182494    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" podUID="3c28b204-2e05-473f-b027-13d086286e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Oct 29 21:52:13.182362 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:13.182331    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-f556s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Oct 29 21:52:13.182546 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:13.182398    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" podUID="3c28b204-2e05-473f-b027-13d086286e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Oct 29 21:52:14.446348 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:14.446315    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:52:14.446822 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:14.446651    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:52:14.447661 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:14.447640    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 21:52:28.410950 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:52:28.410911    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:52:42.526083 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.526048    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"]
Oct 29 21:52:42.526617 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.526440    7082 cpu_manager.go:414] "containerMap: removing container" podUID="7428044b-b88e-4e73-84f8-d367a812703a" containerName="authorino"
Oct 29 21:52:42.526617 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.526452    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7428044b-b88e-4e73-84f8-d367a812703a" containerName="authorino"
Oct 29 21:52:42.526617 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.526545    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="7428044b-b88e-4e73-84f8-d367a812703a" containerName="authorino"
Oct 29 21:52:42.529865 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.529848    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.558806 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.558768    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"]
Oct 29 21:52:42.645345 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.645307    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.645345 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.645345    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.645594 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.645363    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.645594 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.645446    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.645594 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.645476    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.645594 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.645517    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.645594 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.645569    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.645865 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.645689    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.645865 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.645757    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qcjt\" (UniqueName: \"kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-kube-api-access-7qcjt\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.746710 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.746675    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.746922 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.746722    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qcjt\" (UniqueName: \"kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-kube-api-access-7qcjt\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.746922 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.746747    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.746922 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.746764    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.746922 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.746780    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.746922 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.746837    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.746922 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.746855    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.746922 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.746875    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.747290 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.746961    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.747290 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.747201    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.747433 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.747290    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.747433 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.747379    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.747543 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.747441    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.747759 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.747739    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.749317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.749295    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.749731 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.749709    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.769992 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.769967    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.770118 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.770104    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qcjt\" (UniqueName: \"kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-kube-api-access-7qcjt\") pod \"maas-default-gateway-openshift-default-769df7846c-bhd4j\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:42.842226 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:42.842195    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:43.182319 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:43.182295    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"]
Oct 29 21:52:43.186660 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:52:43.186609    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1644588d_6cb4_4f39_ab4a_04e939a89fe6.slice/crio-41617c4f446b5bbc229946f176fb028b02e69e0d2682ff124bafd9fc2a96ee45 WatchSource:0}: Error finding container 41617c4f446b5bbc229946f176fb028b02e69e0d2682ff124bafd9fc2a96ee45: Status 404 returned error can't find the container with id 41617c4f446b5bbc229946f176fb028b02e69e0d2682ff124bafd9fc2a96ee45
Oct 29 21:52:43.190129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:43.190091    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:52:43.190243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:43.190178    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:52:43.190243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:43.190227    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:52:43.581730 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:43.581695    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j" event={"ID":"1644588d-6cb4-4f39-ab4a-04e939a89fe6","Type":"ContainerStarted","Data":"ba50b1f9d11ae1f1988c187c47c1dbb3e6fc705c06d2009883df074a4c324d48"}
Oct 29 21:52:43.582140 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:43.581753    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j" event={"ID":"1644588d-6cb4-4f39-ab4a-04e939a89fe6","Type":"ContainerStarted","Data":"41617c4f446b5bbc229946f176fb028b02e69e0d2682ff124bafd9fc2a96ee45"}
Oct 29 21:52:43.654098 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:43.654038    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j" podStartSLOduration=1.654022372 podStartE2EDuration="1.654022372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:43.641255334 +0000 UTC m=+336.141297633" watchObservedRunningTime="2026-10-29 21:52:43.654022372 +0000 UTC m=+336.154064669"
Oct 29 21:52:43.843117 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:43.843028    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:44.844742 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:44.844705    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bhd4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 29 21:52:44.845203 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:44.844768    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j" podUID="1644588d-6cb4-4f39-ab4a-04e939a89fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 29 21:52:45.842746 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:45.842711    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bhd4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 29 21:52:45.842934 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:45.842765    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j" podUID="1644588d-6cb4-4f39-ab4a-04e939a89fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 29 21:52:46.843568 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:46.843534    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bhd4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 29 21:52:46.843983 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:46.843598    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j" podUID="1644588d-6cb4-4f39-ab4a-04e939a89fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 29 21:52:47.843153 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:47.843118    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bhd4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 29 21:52:47.843339 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:47.843170    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j" podUID="1644588d-6cb4-4f39-ab4a-04e939a89fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 29 21:52:48.842705 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:48.842666    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-bhd4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 29 21:52:48.843094 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:48.842737    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j" podUID="1644588d-6cb4-4f39-ab4a-04e939a89fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 29 21:52:50.468256 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:50.468212    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:50.468764 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:50.468742    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:50.469546 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:52:50.469527    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 21:52:58.411431 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:52:58.411388    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:53:27.688148 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.688069    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"]
Oct 29 21:53:27.691746 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.691727    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.704026 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.703999    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.704157 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.704031    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.704157 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.704058    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.704243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.704184    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.704243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.704216    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww8wk\" (UniqueName: \"kubernetes.io/projected/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-kube-api-access-ww8wk\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.704329 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.704249    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.704431 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.704403    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.704517 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.704494    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.704569 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.704536    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.717260 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.717232    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"]
Oct 29 21:53:27.805430 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.805395    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.805430 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.805428    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.805685 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.805446    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.805685 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.805477    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.805685 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.805496    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww8wk\" (UniqueName: \"kubernetes.io/projected/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-kube-api-access-ww8wk\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.805685 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.805518    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.805685 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.805543    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.805685 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.805567    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.805685 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.805604    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.806067 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.805890    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.806190 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.806170    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.806246 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.806201    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.806246 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.806237    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.806331 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.806291    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.807898 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.807877    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.808124 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.808105    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.830989 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.830943    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:27.831128 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:27.831040    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww8wk\" (UniqueName: \"kubernetes.io/projected/508e1b3c-a7fe-4c6f-bc38-128cec822dc6-kube-api-access-ww8wk\") pod \"maas-default-gateway-openshift-default-769df7846c-9xmcs\" (UID: \"508e1b3c-a7fe-4c6f-bc38-128cec822dc6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:28.006505 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:28.006412    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:28.327342 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:28.327230    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"]
Oct 29 21:53:28.330070 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:53:28.330040    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508e1b3c_a7fe_4c6f_bc38_128cec822dc6.slice/crio-c24225f508ccdd0eb6c1724966fe46c81110f5584b29b80761478913ad6c3f65 WatchSource:0}: Error finding container c24225f508ccdd0eb6c1724966fe46c81110f5584b29b80761478913ad6c3f65: Status 404 returned error can't find the container with id c24225f508ccdd0eb6c1724966fe46c81110f5584b29b80761478913ad6c3f65
Oct 29 21:53:28.332271 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:28.332234    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:53:28.332361 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:28.332311    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:53:28.332361 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:28.332339    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:53:28.415845 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:53:28.415809    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:53:28.835284 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:28.835252    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs" event={"ID":"508e1b3c-a7fe-4c6f-bc38-128cec822dc6","Type":"ContainerStarted","Data":"530ef780ff468cefc0e23e1322ae8b2a63a0f138dcc7e87ef8c75ca2a37a0aa0"}
Oct 29 21:53:28.835691 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:28.835303    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs" event={"ID":"508e1b3c-a7fe-4c6f-bc38-128cec822dc6","Type":"ContainerStarted","Data":"c24225f508ccdd0eb6c1724966fe46c81110f5584b29b80761478913ad6c3f65"}
Oct 29 21:53:28.910985 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:28.910927    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs" podStartSLOduration=1.910909382 podStartE2EDuration="1.910909382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:28.893137915 +0000 UTC m=+381.393180238" watchObservedRunningTime="2026-10-29 21:53:28.910909382 +0000 UTC m=+381.410951675"
Oct 29 21:53:29.009653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:29.009461    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:29.009851 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:29.009672    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-9xmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 29 21:53:29.009851 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:29.009729    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs" podUID="508e1b3c-a7fe-4c6f-bc38-128cec822dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 29 21:53:30.010077 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:30.010037    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-9xmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 29 21:53:30.010489 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:30.010091    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs" podUID="508e1b3c-a7fe-4c6f-bc38-128cec822dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 29 21:53:31.007652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:31.006911    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-9xmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 29 21:53:31.009736 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:31.009683    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs" podUID="508e1b3c-a7fe-4c6f-bc38-128cec822dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 29 21:53:32.011124 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:32.011088    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-9xmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 29 21:53:32.011796 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:32.011751    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs" podUID="508e1b3c-a7fe-4c6f-bc38-128cec822dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 29 21:53:33.008106 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:33.008064    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-9xmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 29 21:53:33.008526 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:33.008490    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs" podUID="508e1b3c-a7fe-4c6f-bc38-128cec822dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 29 21:53:34.008119 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:34.008077    7082 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-9xmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 29 21:53:34.009145 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:34.008727    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs" podUID="508e1b3c-a7fe-4c6f-bc38-128cec822dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 29 21:53:35.010391 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:35.010360    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:35.010887 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:35.010807    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:35.011560 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:35.011543    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9xmcs"
Oct 29 21:53:44.390222 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.390183    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"]
Oct 29 21:53:44.394824 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.394799    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.422520 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.422492    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 29 21:53:44.422520 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.422511    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 29 21:53:44.422744 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.422597    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5z45z\""
Oct 29 21:53:44.422744 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.422726    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 29 21:53:44.424450 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.424434    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"]
Oct 29 21:53:44.464684 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.464646    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f069baec-59d9-4c9e-b5a2-c25261d01cf4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.464884 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.464702    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f069baec-59d9-4c9e-b5a2-c25261d01cf4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.464884 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.464774    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f069baec-59d9-4c9e-b5a2-c25261d01cf4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.464984 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.464880    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f069baec-59d9-4c9e-b5a2-c25261d01cf4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.464984 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.464969    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f069baec-59d9-4c9e-b5a2-c25261d01cf4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.465058 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.465018    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp8pd\" (UniqueName: \"kubernetes.io/projected/f069baec-59d9-4c9e-b5a2-c25261d01cf4-kube-api-access-sp8pd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.566251 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.566216    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f069baec-59d9-4c9e-b5a2-c25261d01cf4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.566430 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.566269    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp8pd\" (UniqueName: \"kubernetes.io/projected/f069baec-59d9-4c9e-b5a2-c25261d01cf4-kube-api-access-sp8pd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.566430 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.566380    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f069baec-59d9-4c9e-b5a2-c25261d01cf4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.566515 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.566430    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f069baec-59d9-4c9e-b5a2-c25261d01cf4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.566515 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.566466    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f069baec-59d9-4c9e-b5a2-c25261d01cf4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.566670 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.566596    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f069baec-59d9-4c9e-b5a2-c25261d01cf4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.566988 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.566905    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f069baec-59d9-4c9e-b5a2-c25261d01cf4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.566988 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.566913    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f069baec-59d9-4c9e-b5a2-c25261d01cf4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.568206 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.568178    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f069baec-59d9-4c9e-b5a2-c25261d01cf4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.568964 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.568936    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f069baec-59d9-4c9e-b5a2-c25261d01cf4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.569285 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.569262    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f069baec-59d9-4c9e-b5a2-c25261d01cf4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.594048 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.594013    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp8pd\" (UniqueName: \"kubernetes.io/projected/f069baec-59d9-4c9e-b5a2-c25261d01cf4-kube-api-access-sp8pd\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt\" (UID: \"f069baec-59d9-4c9e-b5a2-c25261d01cf4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:44.711370 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:44.711286    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:45.025976 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:45.025946    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"]
Oct 29 21:53:45.028785 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:53:45.028758    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf069baec_59d9_4c9e_b5a2_c25261d01cf4.slice/crio-cb02aa47b290b127e832503f68420c47fc46bb27a01eb673f39f5149d8438606 WatchSource:0}: Error finding container cb02aa47b290b127e832503f68420c47fc46bb27a01eb673f39f5149d8438606: Status 404 returned error can't find the container with id cb02aa47b290b127e832503f68420c47fc46bb27a01eb673f39f5149d8438606
Oct 29 21:53:45.537148 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:45.537108    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:53:45.937347 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:45.937269    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt" event={"ID":"f069baec-59d9-4c9e-b5a2-c25261d01cf4","Type":"ContainerStarted","Data":"cb02aa47b290b127e832503f68420c47fc46bb27a01eb673f39f5149d8438606"}
Oct 29 21:53:46.947371 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:46.947284    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt" event={"ID":"f069baec-59d9-4c9e-b5a2-c25261d01cf4","Type":"ContainerStarted","Data":"0b80e7cca1af543fc37aa630ef0f9b93e1dcb26f3ea728ab1d1e840b5ad1c56f"}
Oct 29 21:53:46.992286 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:46.992236    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt" podStartSLOduration=1.365518899 podStartE2EDuration="2.992223491s" totalImagesPullingTime="1.626704592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:46.990640514 +0000 UTC m=+399.490682807" watchObservedRunningTime="2026-10-29 21:53:46.992223491 +0000 UTC m=+399.492265789"
Oct 29 21:53:47.711651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:47.711601    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:50.678009 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:50.677975    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:53:52.622033 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.622001    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"]
Oct 29 21:53:52.631896 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.631869    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.655290 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.653137    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 29 21:53:52.670853 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.670809    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"]
Oct 29 21:53:52.710094 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.710053    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdqh4\" (UniqueName: \"kubernetes.io/projected/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-kube-api-access-rdqh4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.710276 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.710120    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.710276 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.710205    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.710276 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.710248    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.710380 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.710303    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.710380 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.710340    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.811433 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.811393    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.812271 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.812155    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.812502 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.812485    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.812948 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.812849    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.813102 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.813088    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.813915 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.813892    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.814082 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.814058    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.814192 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.814140    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdqh4\" (UniqueName: \"kubernetes.io/projected/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-kube-api-access-rdqh4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.814252 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.814204    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.816792 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.816765    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.818664 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.818230    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.838426 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.838399    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdqh4\" (UniqueName: \"kubernetes.io/projected/0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496-kube-api-access-rdqh4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tbk7p\" (UID: \"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:52.959754 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:52.959664    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:53.273697 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:53.271888    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"]
Oct 29 21:53:53.286533 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:53:53.286501    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc3bfd2_1b15_4edd_ad1f_de9cdf0d7496.slice/crio-033b03ace97a241c01128f7da80bcec3f8692f9a84f7758a92766a01a0d491a6 WatchSource:0}: Error finding container 033b03ace97a241c01128f7da80bcec3f8692f9a84f7758a92766a01a0d491a6: Status 404 returned error can't find the container with id 033b03ace97a241c01128f7da80bcec3f8692f9a84f7758a92766a01a0d491a6
Oct 29 21:53:53.993972 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:53.993932    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" event={"ID":"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496","Type":"ContainerStarted","Data":"909f9eb4aa2e6e374507eb7dfc69744833a1103e7e15eaa05a44edb63d0f8e1e"}
Oct 29 21:53:53.994386 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:53.993999    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" event={"ID":"0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496","Type":"ContainerStarted","Data":"033b03ace97a241c01128f7da80bcec3f8692f9a84f7758a92766a01a0d491a6"}
Oct 29 21:53:54.037446 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:54.037389    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" podStartSLOduration=1.8121465890000001 podStartE2EDuration="2.03737353s" totalImagesPullingTime="225.226941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:54.036716865 +0000 UTC m=+406.536759165" watchObservedRunningTime="2026-10-29 21:53:54.03737353 +0000 UTC m=+406.537415841"
Oct 29 21:53:54.719857 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:54.719822    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:54.737731 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:54.737703    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:54.747132 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:54.747102    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:53:54.960820 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:54.960782    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:53:55.011129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:55.011034    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt"
Oct 29 21:53:57.377672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:57.377615    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:53:57.814225 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:57.812414    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-zqhtx"]
Oct 29 21:53:57.842182 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:57.842142    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:57.870757 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:57.870725    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-zqhtx"]
Oct 29 21:53:57.874080 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:57.871791    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 29 21:53:57.977468 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:57.977424    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4684c6fd-8ac7-4005-a20d-48396c0ab981-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:57.977715 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:57.977549    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4684c6fd-8ac7-4005-a20d-48396c0ab981-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:57.977715 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:57.977586    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4684c6fd-8ac7-4005-a20d-48396c0ab981-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:57.977715 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:57.977691    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4684c6fd-8ac7-4005-a20d-48396c0ab981-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:57.977880 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:57.977724    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4684c6fd-8ac7-4005-a20d-48396c0ab981-home\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:57.977880 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:57.977765    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzx2\" (UniqueName: \"kubernetes.io/projected/4684c6fd-8ac7-4005-a20d-48396c0ab981-kube-api-access-mhzx2\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.078988 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.078903    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4684c6fd-8ac7-4005-a20d-48396c0ab981-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.078988 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.078942    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4684c6fd-8ac7-4005-a20d-48396c0ab981-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.078988 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.078971    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4684c6fd-8ac7-4005-a20d-48396c0ab981-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.079284 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.079084    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4684c6fd-8ac7-4005-a20d-48396c0ab981-home\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.079284 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.079151    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzx2\" (UniqueName: \"kubernetes.io/projected/4684c6fd-8ac7-4005-a20d-48396c0ab981-kube-api-access-mhzx2\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.079284 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.079218    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4684c6fd-8ac7-4005-a20d-48396c0ab981-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.079418 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.079374    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4684c6fd-8ac7-4005-a20d-48396c0ab981-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.079418 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.079379    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4684c6fd-8ac7-4005-a20d-48396c0ab981-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.079607 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.079586    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4684c6fd-8ac7-4005-a20d-48396c0ab981-home\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.081780 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.081753    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4684c6fd-8ac7-4005-a20d-48396c0ab981-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.082085 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.082065    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4684c6fd-8ac7-4005-a20d-48396c0ab981-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.105022 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.104986    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzx2\" (UniqueName: \"kubernetes.io/projected/4684c6fd-8ac7-4005-a20d-48396c0ab981-kube-api-access-mhzx2\") pod \"echo-simulator-kserve-64b6f8f96c-zqhtx\" (UID: \"4684c6fd-8ac7-4005-a20d-48396c0ab981\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.196918 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.196875    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:53:58.431480 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:53:58.431377    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:53:58.540651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:58.539953    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-zqhtx"]
Oct 29 21:53:59.037209 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:53:59.037125    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" event={"ID":"4684c6fd-8ac7-4005-a20d-48396c0ab981","Type":"ContainerStarted","Data":"7f7d851fed105293530ea926dcc7c6886512027a27b7d02311e6d727c042e6d3"}
Oct 29 21:54:00.319674 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.312081    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"]
Oct 29 21:54:00.321552 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.320829    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.355697 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.354389    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 29 21:54:00.357085 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.356983    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"]
Oct 29 21:54:00.474486 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.474443    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.474691 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.474502    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mtbp\" (UniqueName: \"kubernetes.io/projected/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-kube-api-access-6mtbp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.474691 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.474546    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.474691 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.474605    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.474871 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.474726    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.474871 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.474757    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.575997 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.575914    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.575997 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.575986    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mtbp\" (UniqueName: \"kubernetes.io/projected/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-kube-api-access-6mtbp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.576307 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.576034    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.576307 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.576211    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.576423 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.576311    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.576423 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.576334    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.576524 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.576418    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.577188 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.577157    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.577304 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.577236    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.578971 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.578941    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.579105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.579052    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.600447 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.600411    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mtbp\" (UniqueName: \"kubernetes.io/projected/c32b079b-eba6-4e6f-8f79-f9c3cb86de99-kube-api-access-6mtbp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs\" (UID: \"c32b079b-eba6-4e6f-8f79-f9c3cb86de99\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:00.666617 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:00.666581    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:01.006807 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:01.006779    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"]
Oct 29 21:54:01.009098 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:54:01.009063    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32b079b_eba6_4e6f_8f79_f9c3cb86de99.slice/crio-1fa82159004e8bc3ceaaf412fc37575fa44b3b9f4c2991beddb6599ef610ec7d WatchSource:0}: Error finding container 1fa82159004e8bc3ceaaf412fc37575fa44b3b9f4c2991beddb6599ef610ec7d: Status 404 returned error can't find the container with id 1fa82159004e8bc3ceaaf412fc37575fa44b3b9f4c2991beddb6599ef610ec7d
Oct 29 21:54:01.079326 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:01.079274    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs" event={"ID":"c32b079b-eba6-4e6f-8f79-f9c3cb86de99","Type":"ContainerStarted","Data":"1fa82159004e8bc3ceaaf412fc37575fa44b3b9f4c2991beddb6599ef610ec7d"}
Oct 29 21:54:01.701663 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:01.701608    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:54:02.090569 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.088934    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs" event={"ID":"c32b079b-eba6-4e6f-8f79-f9c3cb86de99","Type":"ContainerStarted","Data":"ecdb67ffa7c3ce0551b766c89476ddf1c2feea8caa5569096400f231e74f73b6"}
Oct 29 21:54:02.091433 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.091400    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" event={"ID":"4684c6fd-8ac7-4005-a20d-48396c0ab981","Type":"ContainerStarted","Data":"19a5c0dea9bbad24983042ae9c10273fe4d7a1e9856fb2f6c95c99e2d0b6743b"}
Oct 29 21:54:02.136496 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.136436    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs" podStartSLOduration=1.927433001 podStartE2EDuration="2.136415051s" totalImagesPullingTime="208.98205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:02.134930786 +0000 UTC m=+414.634973090" watchObservedRunningTime="2026-10-29 21:54:02.136415051 +0000 UTC m=+414.636457349"
Oct 29 21:54:02.179650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.175283    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"]
Oct 29 21:54:02.184918 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.184888    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.187765 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.187476    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" podStartSLOduration=2.456597288 podStartE2EDuration="5.18745512s" totalImagesPullingTime="2.730857832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:02.18630688 +0000 UTC m=+414.686349180" watchObservedRunningTime="2026-10-29 21:54:02.18745512 +0000 UTC m=+414.687497420"
Oct 29 21:54:02.194274 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.194238    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb1b9db-8f2e-448e-8b42-96025377751b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.194522 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.194504    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7sjx\" (UniqueName: \"kubernetes.io/projected/2eb1b9db-8f2e-448e-8b42-96025377751b-kube-api-access-f7sjx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.194661 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.194644    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb1b9db-8f2e-448e-8b42-96025377751b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.194782 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.194768    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb1b9db-8f2e-448e-8b42-96025377751b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.194903 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.194889    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb1b9db-8f2e-448e-8b42-96025377751b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.195071 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.195052    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9db-8f2e-448e-8b42-96025377751b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.205903 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.205877    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 29 21:54:02.208419 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.208397    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:54:02.213164 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.213141    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"]
Oct 29 21:54:02.296578 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.296526    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9db-8f2e-448e-8b42-96025377751b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.296774 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.296681    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb1b9db-8f2e-448e-8b42-96025377751b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.296774 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.296728    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7sjx\" (UniqueName: \"kubernetes.io/projected/2eb1b9db-8f2e-448e-8b42-96025377751b-kube-api-access-f7sjx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.296774 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.296762    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb1b9db-8f2e-448e-8b42-96025377751b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.296933 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.296805    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb1b9db-8f2e-448e-8b42-96025377751b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.296933 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.296839    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb1b9db-8f2e-448e-8b42-96025377751b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.297403 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.297361    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb1b9db-8f2e-448e-8b42-96025377751b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.297668 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.297425    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb1b9db-8f2e-448e-8b42-96025377751b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.297777 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.297697    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb1b9db-8f2e-448e-8b42-96025377751b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.299619 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.299595    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb1b9db-8f2e-448e-8b42-96025377751b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.300074 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.300056    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb1b9db-8f2e-448e-8b42-96025377751b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.321798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.321755    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7sjx\" (UniqueName: \"kubernetes.io/projected/2eb1b9db-8f2e-448e-8b42-96025377751b-kube-api-access-f7sjx\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q\" (UID: \"2eb1b9db-8f2e-448e-8b42-96025377751b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.518102 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.518001    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:02.667117 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.667086    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:02.835358 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.835196    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"]
Oct 29 21:54:02.841370 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:54:02.841326    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2eb1b9db_8f2e_448e_8b42_96025377751b.slice/crio-4feed159814d12e629f470d75980517262e557627771face5f2066ed66225616 WatchSource:0}: Error finding container 4feed159814d12e629f470d75980517262e557627771face5f2066ed66225616: Status 404 returned error can't find the container with id 4feed159814d12e629f470d75980517262e557627771face5f2066ed66225616
Oct 29 21:54:02.960004 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.959944    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:54:02.979199 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:02.979168    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:54:03.110583 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:03.110544    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" event={"ID":"2eb1b9db-8f2e-448e-8b42-96025377751b","Type":"ContainerStarted","Data":"4feed159814d12e629f470d75980517262e557627771face5f2066ed66225616"}
Oct 29 21:54:03.131034 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:03.131002    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 21:54:04.154650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:04.150439    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" event={"ID":"2eb1b9db-8f2e-448e-8b42-96025377751b","Type":"ContainerStarted","Data":"83404bc8d8eb9669a8fb9a1de46f71ddc1f4177ac56f9fb5956b253758bdbf3c"}
Oct 29 21:54:04.202653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:04.202063    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podStartSLOduration=1.9770149830000001 podStartE2EDuration="2.202043968s" totalImagesPullingTime="225.028985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:04.199038171 +0000 UTC m=+416.699080471" watchObservedRunningTime="2026-10-29 21:54:04.202043968 +0000 UTC m=+416.702086268"
Oct 29 21:54:04.522702 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:04.521743    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:08.198173 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:08.198139    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:54:08.210422 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:08.210399    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:54:09.192142 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:09.192109    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 21:54:09.368464 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:09.368427    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:54:10.667076 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:10.667045    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:10.679395 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:10.679371    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:11.204221 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:11.204191    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs"
Oct 29 21:54:12.519055 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:12.519010    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:12.532654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:12.532615    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:13.216060 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:13.216021    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:54:18.962686 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:18.962649    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8468f79957-dxp8f"]
Oct 29 21:54:18.966565 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:18.966545    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8468f79957-dxp8f"
Oct 29 21:54:18.986157 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:18.986131    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8468f79957-dxp8f"]
Oct 29 21:54:19.013723 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:19.013697    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c-tls-cert\") pod \"authorino-8468f79957-dxp8f\" (UID: \"fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c\") " pod="kuadrant-system/authorino-8468f79957-dxp8f"
Oct 29 21:54:19.013841 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:19.013736    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6rh\" (UniqueName: \"kubernetes.io/projected/fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c-kube-api-access-gn6rh\") pod \"authorino-8468f79957-dxp8f\" (UID: \"fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c\") " pod="kuadrant-system/authorino-8468f79957-dxp8f"
Oct 29 21:54:19.114738 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:19.114714    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c-tls-cert\") pod \"authorino-8468f79957-dxp8f\" (UID: \"fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c\") " pod="kuadrant-system/authorino-8468f79957-dxp8f"
Oct 29 21:54:19.114896 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:19.114753    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6rh\" (UniqueName: \"kubernetes.io/projected/fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c-kube-api-access-gn6rh\") pod \"authorino-8468f79957-dxp8f\" (UID: \"fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c\") " pod="kuadrant-system/authorino-8468f79957-dxp8f"
Oct 29 21:54:19.117303 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:19.117282    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c-tls-cert\") pod \"authorino-8468f79957-dxp8f\" (UID: \"fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c\") " pod="kuadrant-system/authorino-8468f79957-dxp8f"
Oct 29 21:54:19.138205 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:19.138179    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6rh\" (UniqueName: \"kubernetes.io/projected/fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c-kube-api-access-gn6rh\") pod \"authorino-8468f79957-dxp8f\" (UID: \"fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c\") " pod="kuadrant-system/authorino-8468f79957-dxp8f"
Oct 29 21:54:19.278553 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:19.278476    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8468f79957-dxp8f"
Oct 29 21:54:19.579926 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:19.579903    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8468f79957-dxp8f"]
Oct 29 21:54:19.582115 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:54:19.582088    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc83c5e9_bbc9_4365_bbc0_1068b5c44c8c.slice/crio-2d489728f8d3061407693083225eb58b54297a37f18b046e99aee62e36efa970 WatchSource:0}: Error finding container 2d489728f8d3061407693083225eb58b54297a37f18b046e99aee62e36efa970: Status 404 returned error can't find the container with id 2d489728f8d3061407693083225eb58b54297a37f18b046e99aee62e36efa970
Oct 29 21:54:20.246855 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:20.246812    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8468f79957-dxp8f" event={"ID":"fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c","Type":"ContainerStarted","Data":"2d489728f8d3061407693083225eb58b54297a37f18b046e99aee62e36efa970"}
Oct 29 21:54:21.253395 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:21.253350    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8468f79957-dxp8f" event={"ID":"fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c","Type":"ContainerStarted","Data":"3b0233d2e784b272990c7faca150cec9b84dda34e1c97ab2b79623727ca4e62a"}
Oct 29 21:54:21.294684 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:21.294615    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-8468f79957-dxp8f" podStartSLOduration=2.455708181 podStartE2EDuration="3.29460138s" totalImagesPullingTime="838.893199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:21.293093674 +0000 UTC m=+433.793135974" watchObservedRunningTime="2026-10-29 21:54:21.29460138 +0000 UTC m=+433.794643730"
Oct 29 21:54:21.326267 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:21.326226    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8687494d7b-rpm8k"]
Oct 29 21:54:21.326951 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:21.326597    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-8687494d7b-rpm8k" podUID="3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2" containerName="authorino" containerID="cri-o://299650874af8335f64c7d7a11fbee74904e7652291e6619da6e95b047055aa74" gracePeriod=30
Oct 29 21:54:21.611538 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:21.611510    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8687494d7b-rpm8k"
Oct 29 21:54:21.742993 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:21.742961    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-tls-cert\" (UniqueName: \"kubernetes.io/secret/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-tls-cert\") pod \"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2\" (UID: \"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2\") "
Oct 29 21:54:21.743200 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:21.743116    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-kube-api-access-rqt2h\" (UniqueName: \"kubernetes.io/projected/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-kube-api-access-rqt2h\") pod \"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2\" (UID: \"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2\") "
Oct 29 21:54:21.745915 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:21.745879    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-kube-api-access-rqt2h" pod "3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2" (UID: "3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2"). InnerVolumeSpecName "kube-api-access-rqt2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:54:21.756469 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:21.756440    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-tls-cert" pod "3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2" (UID: "3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:54:21.844606 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:21.844570    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqt2h\" (UniqueName: \"kubernetes.io/projected/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-kube-api-access-rqt2h\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:54:21.844606 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:21.844601    7082 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2-tls-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:54:22.263553 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:22.263459    7082 generic.go:356] "Generic (PLEG): container finished" podID="3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2" containerID="299650874af8335f64c7d7a11fbee74904e7652291e6619da6e95b047055aa74" exitCode=0
Oct 29 21:54:22.263553 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:22.263515    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8687494d7b-rpm8k"
Oct 29 21:54:22.263553 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:22.263545    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8687494d7b-rpm8k" event={"ID":"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2","Type":"ContainerDied","Data":"299650874af8335f64c7d7a11fbee74904e7652291e6619da6e95b047055aa74"}
Oct 29 21:54:22.264161 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:22.263584    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8687494d7b-rpm8k" event={"ID":"3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2","Type":"ContainerDied","Data":"2241dd442a0c9703b5437d8e558bf1fa02bf86993ebd6f3900bfcd08136cd668"}
Oct 29 21:54:22.264161 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:22.263609    7082 scope.go:122] "RemoveContainer" containerID="299650874af8335f64c7d7a11fbee74904e7652291e6619da6e95b047055aa74"
Oct 29 21:54:22.274835 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:22.274817    7082 scope.go:122] "RemoveContainer" containerID="299650874af8335f64c7d7a11fbee74904e7652291e6619da6e95b047055aa74"
Oct 29 21:54:22.275116 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:54:22.275090    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299650874af8335f64c7d7a11fbee74904e7652291e6619da6e95b047055aa74\": container with ID starting with 299650874af8335f64c7d7a11fbee74904e7652291e6619da6e95b047055aa74 not found: ID does not exist" containerID="299650874af8335f64c7d7a11fbee74904e7652291e6619da6e95b047055aa74"
Oct 29 21:54:22.275185 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:22.275119    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299650874af8335f64c7d7a11fbee74904e7652291e6619da6e95b047055aa74"} err="failed to get container status \"299650874af8335f64c7d7a11fbee74904e7652291e6619da6e95b047055aa74\": rpc error: code = NotFound desc = could not find container \"299650874af8335f64c7d7a11fbee74904e7652291e6619da6e95b047055aa74\": container with ID starting with 299650874af8335f64c7d7a11fbee74904e7652291e6619da6e95b047055aa74 not found: ID does not exist"
Oct 29 21:54:22.322417 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:22.322382    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8687494d7b-rpm8k"]
Oct 29 21:54:22.326658 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:22.326618    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8687494d7b-rpm8k"]
Oct 29 21:54:23.630398 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:23.630365    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2" path="/var/lib/kubelet/pods/3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2/volumes"
Oct 29 21:54:28.411690 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:54:28.411615    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:54:58.411829 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:54:58.411781    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:54:59.128668 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.127218    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"]
Oct 29 21:54:59.128668 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.127806    7082 cpu_manager.go:414] "containerMap: removing container" podUID="3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2" containerName="authorino"
Oct 29 21:54:59.128668 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.127820    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2" containerName="authorino"
Oct 29 21:54:59.128668 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.127897    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1f5993-efcc-42f8-9b47-0e0e4e9f98d2" containerName="authorino"
Oct 29 21:54:59.131716 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.131696    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.150903 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.150878    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-de7186ef-openshift-default-dockercfg-pvmqn\""
Oct 29 21:54:59.165227 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.165198    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"]
Oct 29 21:54:59.213316 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.213284    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-socket\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.213461 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.213320    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-certs\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.213461 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.213344    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-data\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.213534 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.213463    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhmlf\" (UniqueName: \"kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-kube-api-access-fhmlf\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.213567 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.213539    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05954176-b158-4666-933b-334e368ef303-istiod-ca-cert\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.213621 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.213605    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-envoy\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.213683 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.213666    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05954176-b158-4666-933b-334e368ef303-istio-podinfo\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.213752 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.213735    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-istio-token\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.213828 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.213816    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-credential-socket\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.315254 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.315222    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-socket\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.315435 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.315260    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-certs\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.315435 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.315284    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-data\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.315435 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.315301    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhmlf\" (UniqueName: \"kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-kube-api-access-fhmlf\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.315435 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.315342    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05954176-b158-4666-933b-334e368ef303-istiod-ca-cert\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.315435 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.315359    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-envoy\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.315435 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.315377    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05954176-b158-4666-933b-334e368ef303-istio-podinfo\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.315835 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.315529    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-istio-token\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.315835 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.315678    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-credential-socket\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.315835 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.315747    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-certs\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.315835 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.315785    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-socket\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.316039 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.316000    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-data\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.316095 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.316074    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-credential-socket\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.316215 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.316190    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05954176-b158-4666-933b-334e368ef303-istiod-ca-cert\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.317934 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.317906    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-envoy\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.318024 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.317918    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05954176-b158-4666-933b-334e368ef303-istio-podinfo\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.342237 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.342206    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhmlf\" (UniqueName: \"kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-kube-api-access-fhmlf\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.342366 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.342270    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-istio-token\") pod \"e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv\" (UID: \"05954176-b158-4666-933b-334e368ef303\") " pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.400401 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.400327    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"]
Oct 29 21:54:59.405790 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.405767    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.420243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.420219    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f97668b9-openshift-default-dockercfg-q4t2b\""
Oct 29 21:54:59.428037 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.427999    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"]
Oct 29 21:54:59.446615 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.446583    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:54:59.518077 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.518036    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-credential-socket\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.518236 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.518113    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-certs\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.518236 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.518143    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-podinfo\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.518848 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.518222    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-token\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.518944 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.518857    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-socket\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.518944 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.518922    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-data\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.519026 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.518971    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq4rj\" (UniqueName: \"kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-kube-api-access-hq4rj\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.519064 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.519015    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istiod-ca-cert\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.519125 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.519108    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-envoy\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.620041    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-envoy\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.620100    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-credential-socket\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.620133    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-certs\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.620148    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-podinfo\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.620222    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-token\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.620248    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-socket\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.620333    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-data\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.620392    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq4rj\" (UniqueName: \"kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-kube-api-access-hq4rj\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.620434    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istiod-ca-cert\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.620561    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-credential-socket\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.620859    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-certs\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.620977    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-data\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.621150    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istiod-ca-cert\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.621407    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-socket\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.623676    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-envoy\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.630317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.624714    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-podinfo\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.644387 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.644358    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-token\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.644506 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.644482    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq4rj\" (UniqueName: \"kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-kube-api-access-hq4rj\") pod \"e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") " pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.719658 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.719566    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:54:59.773352 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.773311    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"]
Oct 29 21:54:59.793149 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.793110    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:54:59.793268 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.793193    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:54:59.793268 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:54:59.793236    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:00.042715 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:00.042596    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"]
Oct 29 21:55:00.050231 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:00.050196    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bb9e0c_17cf_41f8_b79e_841fa270c02b.slice/crio-3076f9b5ba20594db6b0c32c4d39ccc6471a2a01d4b30b6aa0e93208c8de41bb WatchSource:0}: Error finding container 3076f9b5ba20594db6b0c32c4d39ccc6471a2a01d4b30b6aa0e93208c8de41bb: Status 404 returned error can't find the container with id 3076f9b5ba20594db6b0c32c4d39ccc6471a2a01d4b30b6aa0e93208c8de41bb
Oct 29 21:55:00.054940 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:00.054898    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:00.055196 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:00.055161    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:00.055314 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:00.055262    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:00.492350 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:00.492315    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" event={"ID":"45bb9e0c-17cf-41f8-b79e-841fa270c02b","Type":"ContainerStarted","Data":"5d9403b6be3b05c4cff055788b4b413ba89d028bed746bd7426e56322f24983f"}
Oct 29 21:55:00.493125 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:00.492376    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" event={"ID":"45bb9e0c-17cf-41f8-b79e-841fa270c02b","Type":"ContainerStarted","Data":"3076f9b5ba20594db6b0c32c4d39ccc6471a2a01d4b30b6aa0e93208c8de41bb"}
Oct 29 21:55:00.493966 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:00.493946    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" event={"ID":"05954176-b158-4666-933b-334e368ef303","Type":"ContainerStarted","Data":"c9d21e6d3dde9c1560cd50b67969b3838a9b808d9b518f976cc6382e5779bd90"}
Oct 29 21:55:00.494049 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:00.493989    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" event={"ID":"05954176-b158-4666-933b-334e368ef303","Type":"ContainerStarted","Data":"026b9c9ddc8b74d88cbbf7be28a589665a2a2ec1e29dd7efda38ede9572f61b1"}
Oct 29 21:55:00.541328 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:00.541279    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" podStartSLOduration=1.54126639 podStartE2EDuration="1.54126639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:00.539949189 +0000 UTC m=+473.039991489" watchObservedRunningTime="2026-10-29 21:55:00.54126639 +0000 UTC m=+473.041308690"
Oct 29 21:55:00.588687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:00.588523    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" podStartSLOduration=1.58850646 podStartE2EDuration="1.58850646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:00.584989266 +0000 UTC m=+473.085031566" watchObservedRunningTime="2026-10-29 21:55:00.58850646 +0000 UTC m=+473.088548760"
Oct 29 21:55:00.720136 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:00.720102    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:55:01.026359 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.026318    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"]
Oct 29 21:55:01.030862 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.030839    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.056718 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.056681    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"]
Oct 29 21:55:01.123775 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.123741    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e4e24e-openshift-default-dockercfg-9t52r\""
Oct 29 21:55:01.138318 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.138283    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwjdr\" (UniqueName: \"kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-kube-api-access-gwjdr\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.138556 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.138388    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.138556 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.138452    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.138556 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.138494    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.138556 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.138546    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.138829 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.138573    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.138829 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.138593    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.138829 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.138610    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.138986 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.138823    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.240341 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.240306    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.240541 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.240361    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.240541 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.240393    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.240541 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.240436    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.240769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.240569    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.240769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.240614    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.240769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.240673    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.240769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.240697    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.240769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.240768    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjdr\" (UniqueName: \"kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-kube-api-access-gwjdr\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.241008 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.240946    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.241008 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.240985    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.241114 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.241027    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.241235 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.241219    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.241304 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.241283    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.243233 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.243205    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.243340 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.243281    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.264528 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.264505    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.264680 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.264575    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwjdr\" (UniqueName: \"kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-kube-api-access-gwjdr\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.346023 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.345989    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:01.447266 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.447230    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:55:01.453152 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.453125    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:55:01.501035 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.501004    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:55:01.502123 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.502103    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:55:01.682003 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:01.680655    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3feeed_2094_44d9_aad9_00d17e6bbef0.slice/crio-d597a9a6faa68fbdf4af4892f99b93b3ebb6cabc6142f987f409079b6b8462de WatchSource:0}: Error finding container d597a9a6faa68fbdf4af4892f99b93b3ebb6cabc6142f987f409079b6b8462de: Status 404 returned error can't find the container with id d597a9a6faa68fbdf4af4892f99b93b3ebb6cabc6142f987f409079b6b8462de
Oct 29 21:55:01.682003 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.681294    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"]
Oct 29 21:55:01.685468 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.685422    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:01.685599 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.685518    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:01.685599 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.685571    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:01.725053 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:01.725030    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:55:02.511256 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:02.511216    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" event={"ID":"ad3feeed-2094-44d9-aad9-00d17e6bbef0","Type":"ContainerStarted","Data":"fc03b2c343d00572920143694606e317b96698aae6680b9a769738f78870d558"}
Oct 29 21:55:02.511820 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:02.511292    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" event={"ID":"ad3feeed-2094-44d9-aad9-00d17e6bbef0","Type":"ContainerStarted","Data":"d597a9a6faa68fbdf4af4892f99b93b3ebb6cabc6142f987f409079b6b8462de"}
Oct 29 21:55:02.511995 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:02.511969    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:55:02.513014 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:02.512995    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:55:02.582685 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:02.582606    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podStartSLOduration=2.582592388 podStartE2EDuration="2.582592388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:02.570102896 +0000 UTC m=+475.070145196" watchObservedRunningTime="2026-10-29 21:55:02.582592388 +0000 UTC m=+475.082634687"
Oct 29 21:55:03.346955 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:03.346917    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:03.351955 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:03.351931    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:03.516476 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:03.516442    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:03.517385 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:03.517370    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:05.740808 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:05.740776    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"]
Oct 29 21:55:05.744757 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:05.744739    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"
Oct 29 21:55:05.760571 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:05.760548    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-64505fc5\""
Oct 29 21:55:05.772160 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:05.772134    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-64505fc5-dockercfg-fsrr5\""
Oct 29 21:55:05.775807 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:05.775785    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"]
Oct 29 21:55:05.794381 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:05.794321    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89951ee0-b7e8-48dd-8441-4da86c0fbc62-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz\" (UID: \"89951ee0-b7e8-48dd-8441-4da86c0fbc62\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"
Oct 29 21:55:05.795508 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:05.794936    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tx2k\" (UniqueName: \"kubernetes.io/projected/89951ee0-b7e8-48dd-8441-4da86c0fbc62-kube-api-access-7tx2k\") pod \"payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz\" (UID: \"89951ee0-b7e8-48dd-8441-4da86c0fbc62\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"
Oct 29 21:55:05.910746 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:05.910709    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89951ee0-b7e8-48dd-8441-4da86c0fbc62-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz\" (UID: \"89951ee0-b7e8-48dd-8441-4da86c0fbc62\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"
Oct 29 21:55:05.910939 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:05.910811    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tx2k\" (UniqueName: \"kubernetes.io/projected/89951ee0-b7e8-48dd-8441-4da86c0fbc62-kube-api-access-7tx2k\") pod \"payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz\" (UID: \"89951ee0-b7e8-48dd-8441-4da86c0fbc62\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"
Oct 29 21:55:05.911493 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:05.911475    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89951ee0-b7e8-48dd-8441-4da86c0fbc62-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz\" (UID: \"89951ee0-b7e8-48dd-8441-4da86c0fbc62\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"
Oct 29 21:55:05.935062 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:05.935036    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tx2k\" (UniqueName: \"kubernetes.io/projected/89951ee0-b7e8-48dd-8441-4da86c0fbc62-kube-api-access-7tx2k\") pod \"payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz\" (UID: \"89951ee0-b7e8-48dd-8441-4da86c0fbc62\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"
Oct 29 21:55:06.058753 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.058724    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"
Oct 29 21:55:06.370121 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.370096    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"]
Oct 29 21:55:06.372479 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:06.372445    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89951ee0_b7e8_48dd_8441_4da86c0fbc62.slice/crio-9d9e979301927f363b175873b91e5c6c58457bcab50dfd22ca94051fe46052be WatchSource:0}: Error finding container 9d9e979301927f363b175873b91e5c6c58457bcab50dfd22ca94051fe46052be: Status 404 returned error can't find the container with id 9d9e979301927f363b175873b91e5c6c58457bcab50dfd22ca94051fe46052be
Oct 29 21:55:06.374816 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.374795    7082 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 21:55:06.536348 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.536308    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz" event={"ID":"89951ee0-b7e8-48dd-8441-4da86c0fbc62","Type":"ContainerStarted","Data":"9d9e979301927f363b175873b91e5c6c58457bcab50dfd22ca94051fe46052be"}
Oct 29 21:55:06.713232 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.713151    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"]
Oct 29 21:55:06.718271 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.718247    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"
Oct 29 21:55:06.734914 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.734893    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-de7186ef\""
Oct 29 21:55:06.735082 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.735065    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-de7186ef-dockercfg-zrflk\""
Oct 29 21:55:06.749476 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.749451    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"]
Oct 29 21:55:06.790335 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.790293    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"]
Oct 29 21:55:06.794570 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.794543    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"
Oct 29 21:55:06.835651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.826017    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd56092-21a8-45fa-9cac-9a6094fc5a50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct\" (UID: \"efd56092-21a8-45fa-9cac-9a6094fc5a50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"
Oct 29 21:55:06.835651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.826128    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zct4w\" (UniqueName: \"kubernetes.io/projected/efd56092-21a8-45fa-9cac-9a6094fc5a50-kube-api-access-zct4w\") pod \"payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct\" (UID: \"efd56092-21a8-45fa-9cac-9a6094fc5a50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"
Oct 29 21:55:06.835651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.826752    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"]
Oct 29 21:55:06.927073 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.927037    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd56092-21a8-45fa-9cac-9a6094fc5a50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct\" (UID: \"efd56092-21a8-45fa-9cac-9a6094fc5a50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"
Oct 29 21:55:06.927267 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.927091    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97afdabd-be6a-4a98-ad49-77909430ed98-plugins-config-volume\") pod \"payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5\" (UID: \"97afdabd-be6a-4a98-ad49-77909430ed98\") " pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"
Oct 29 21:55:06.927267 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.927162    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zct4w\" (UniqueName: \"kubernetes.io/projected/efd56092-21a8-45fa-9cac-9a6094fc5a50-kube-api-access-zct4w\") pod \"payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct\" (UID: \"efd56092-21a8-45fa-9cac-9a6094fc5a50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"
Oct 29 21:55:06.927267 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.927212    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctmlm\" (UniqueName: \"kubernetes.io/projected/97afdabd-be6a-4a98-ad49-77909430ed98-kube-api-access-ctmlm\") pod \"payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5\" (UID: \"97afdabd-be6a-4a98-ad49-77909430ed98\") " pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"
Oct 29 21:55:06.927994 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.927966    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd56092-21a8-45fa-9cac-9a6094fc5a50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct\" (UID: \"efd56092-21a8-45fa-9cac-9a6094fc5a50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"
Oct 29 21:55:06.956839 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:06.956800    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zct4w\" (UniqueName: \"kubernetes.io/projected/efd56092-21a8-45fa-9cac-9a6094fc5a50-kube-api-access-zct4w\") pod \"payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct\" (UID: \"efd56092-21a8-45fa-9cac-9a6094fc5a50\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"
Oct 29 21:55:07.028453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.028365    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97afdabd-be6a-4a98-ad49-77909430ed98-plugins-config-volume\") pod \"payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5\" (UID: \"97afdabd-be6a-4a98-ad49-77909430ed98\") " pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"
Oct 29 21:55:07.028453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.028451    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctmlm\" (UniqueName: \"kubernetes.io/projected/97afdabd-be6a-4a98-ad49-77909430ed98-kube-api-access-ctmlm\") pod \"payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5\" (UID: \"97afdabd-be6a-4a98-ad49-77909430ed98\") " pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"
Oct 29 21:55:07.029928 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.029898    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97afdabd-be6a-4a98-ad49-77909430ed98-plugins-config-volume\") pod \"payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5\" (UID: \"97afdabd-be6a-4a98-ad49-77909430ed98\") " pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"
Oct 29 21:55:07.032324 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.032094    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"
Oct 29 21:55:07.054798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.054764    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctmlm\" (UniqueName: \"kubernetes.io/projected/97afdabd-be6a-4a98-ad49-77909430ed98-kube-api-access-ctmlm\") pod \"payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5\" (UID: \"97afdabd-be6a-4a98-ad49-77909430ed98\") " pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"
Oct 29 21:55:07.134334 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.134302    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"
Oct 29 21:55:07.363965 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.363938    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"]
Oct 29 21:55:07.365119 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:07.365073    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd56092_21a8_45fa_9cac_9a6094fc5a50.slice/crio-7bc2da2f08dc1f46e3b2ed2c9074c43153509e6ab741c7bf9550ae5ad3f32f1e WatchSource:0}: Error finding container 7bc2da2f08dc1f46e3b2ed2c9074c43153509e6ab741c7bf9550ae5ad3f32f1e: Status 404 returned error can't find the container with id 7bc2da2f08dc1f46e3b2ed2c9074c43153509e6ab741c7bf9550ae5ad3f32f1e
Oct 29 21:55:07.456598 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.456570    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"]
Oct 29 21:55:07.458902 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:07.458868    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97afdabd_be6a_4a98_ad49_77909430ed98.slice/crio-01f3a58ca71abcf1f817b82ce04acc51355b2e281eb9caefae1a70abc2ba02a6 WatchSource:0}: Error finding container 01f3a58ca71abcf1f817b82ce04acc51355b2e281eb9caefae1a70abc2ba02a6: Status 404 returned error can't find the container with id 01f3a58ca71abcf1f817b82ce04acc51355b2e281eb9caefae1a70abc2ba02a6
Oct 29 21:55:07.487706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.487678    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"]
Oct 29 21:55:07.493186 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.493158    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"
Oct 29 21:55:07.509742 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.509267    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8fba532a\""
Oct 29 21:55:07.512723 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.512703    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"]
Oct 29 21:55:07.543402 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.543363    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5" event={"ID":"97afdabd-be6a-4a98-ad49-77909430ed98","Type":"ContainerStarted","Data":"01f3a58ca71abcf1f817b82ce04acc51355b2e281eb9caefae1a70abc2ba02a6"}
Oct 29 21:55:07.545057 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.545025    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct" event={"ID":"efd56092-21a8-45fa-9cac-9a6094fc5a50","Type":"ContainerStarted","Data":"7bc2da2f08dc1f46e3b2ed2c9074c43153509e6ab741c7bf9550ae5ad3f32f1e"}
Oct 29 21:55:07.562846 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.562821    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"]
Oct 29 21:55:07.566946 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.566924    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"
Oct 29 21:55:07.586016 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.585991    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8fba532a-dockercfg-sxbvm\""
Oct 29 21:55:07.588676 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.588580    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"]
Oct 29 21:55:07.635902 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.635819    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e80dce6-46ef-46c9-84cd-677cd9e618f0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g\" (UID: \"9e80dce6-46ef-46c9-84cd-677cd9e618f0\") " pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"
Oct 29 21:55:07.636058 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.635904    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcxsf\" (UniqueName: \"kubernetes.io/projected/9e80dce6-46ef-46c9-84cd-677cd9e618f0-kube-api-access-lcxsf\") pod \"payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g\" (UID: \"9e80dce6-46ef-46c9-84cd-677cd9e618f0\") " pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"
Oct 29 21:55:07.737675 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.737614    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzvq\" (UniqueName: \"kubernetes.io/projected/12f60c90-fa0c-4d19-9e5f-42c768a911f4-kube-api-access-kbzvq\") pod \"payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx\" (UID: \"12f60c90-fa0c-4d19-9e5f-42c768a911f4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"
Oct 29 21:55:07.737910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.737739    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e80dce6-46ef-46c9-84cd-677cd9e618f0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g\" (UID: \"9e80dce6-46ef-46c9-84cd-677cd9e618f0\") " pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"
Oct 29 21:55:07.737910 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.737824    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxsf\" (UniqueName: \"kubernetes.io/projected/9e80dce6-46ef-46c9-84cd-677cd9e618f0-kube-api-access-lcxsf\") pod \"payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g\" (UID: \"9e80dce6-46ef-46c9-84cd-677cd9e618f0\") " pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"
Oct 29 21:55:07.738076 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.737937    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f60c90-fa0c-4d19-9e5f-42c768a911f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx\" (UID: \"12f60c90-fa0c-4d19-9e5f-42c768a911f4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"
Oct 29 21:55:07.756420 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.751935    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8fba532a\""
Oct 29 21:55:07.766082 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.766041    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e80dce6-46ef-46c9-84cd-677cd9e618f0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g\" (UID: \"9e80dce6-46ef-46c9-84cd-677cd9e618f0\") " pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"
Oct 29 21:55:07.785667 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.785530    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxsf\" (UniqueName: \"kubernetes.io/projected/9e80dce6-46ef-46c9-84cd-677cd9e618f0-kube-api-access-lcxsf\") pod \"payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g\" (UID: \"9e80dce6-46ef-46c9-84cd-677cd9e618f0\") " pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"
Oct 29 21:55:07.811213 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.810818    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"
Oct 29 21:55:07.840129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.840100    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzvq\" (UniqueName: \"kubernetes.io/projected/12f60c90-fa0c-4d19-9e5f-42c768a911f4-kube-api-access-kbzvq\") pod \"payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx\" (UID: \"12f60c90-fa0c-4d19-9e5f-42c768a911f4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"
Oct 29 21:55:07.840361 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.840338    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f60c90-fa0c-4d19-9e5f-42c768a911f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx\" (UID: \"12f60c90-fa0c-4d19-9e5f-42c768a911f4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"
Oct 29 21:55:07.841282 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.841258    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f60c90-fa0c-4d19-9e5f-42c768a911f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx\" (UID: \"12f60c90-fa0c-4d19-9e5f-42c768a911f4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"
Oct 29 21:55:07.866364 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.866323    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzvq\" (UniqueName: \"kubernetes.io/projected/12f60c90-fa0c-4d19-9e5f-42c768a911f4-kube-api-access-kbzvq\") pod \"payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx\" (UID: \"12f60c90-fa0c-4d19-9e5f-42c768a911f4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"
Oct 29 21:55:07.884317 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:07.884203    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"
Oct 29 21:55:08.179547 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:08.179513    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"]
Oct 29 21:55:08.180421 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:08.180395    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e80dce6_46ef_46c9_84cd_677cd9e618f0.slice/crio-1c5111376979c599d0edf8a6e90921135f7b202ee10efc4a8772fe894e7d6d2c WatchSource:0}: Error finding container 1c5111376979c599d0edf8a6e90921135f7b202ee10efc4a8772fe894e7d6d2c: Status 404 returned error can't find the container with id 1c5111376979c599d0edf8a6e90921135f7b202ee10efc4a8772fe894e7d6d2c
Oct 29 21:55:08.291518 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:08.291465    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"]
Oct 29 21:55:08.557040 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:08.557004    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g" event={"ID":"9e80dce6-46ef-46c9-84cd-677cd9e618f0","Type":"ContainerStarted","Data":"1c5111376979c599d0edf8a6e90921135f7b202ee10efc4a8772fe894e7d6d2c"}
Oct 29 21:55:08.558916 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:08.558885    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx" event={"ID":"12f60c90-fa0c-4d19-9e5f-42c768a911f4","Type":"ContainerStarted","Data":"7ad5b1c8082f1c91aac1dd3c87eac770bd4f6fd9c166d8384e1596b2a40b8bf7"}
Oct 29 21:55:09.039104 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.039021    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"]
Oct 29 21:55:09.054231 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.053359    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.062584 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.053260    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"]
Oct 29 21:55:09.072999 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.072972    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.073243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.073005    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.073243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.073021    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.073243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.073040    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.073243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.073066    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddab3d72-73bf-4033-be01-6177dddff66a-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.073243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.073090    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.073243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.073103    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddab3d72-73bf-4033-be01-6177dddff66a-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.073243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.073121    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.073657 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.073319    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpqxq\" (UniqueName: \"kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-kube-api-access-wpqxq\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.073657 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.073415    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a3c02e-openshift-default-dockercfg-tvljv\""
Oct 29 21:55:09.179402 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.179365    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.179402 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.179407    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.179684 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.179422    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.179684 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.179442    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.179684 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.179473    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddab3d72-73bf-4033-be01-6177dddff66a-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.179684 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.179500    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.179684 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.179515    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddab3d72-73bf-4033-be01-6177dddff66a-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.179684 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.179531    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.179684 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.179557    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpqxq\" (UniqueName: \"kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-kube-api-access-wpqxq\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.180786 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.180077    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.180786 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.180163    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.180786 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.180475    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.180786 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.180527    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.180786 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.180687    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddab3d72-73bf-4033-be01-6177dddff66a-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.189182 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.184651    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddab3d72-73bf-4033-be01-6177dddff66a-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.198913 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.198879    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.208862 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.208824    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpqxq\" (UniqueName: \"kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-kube-api-access-wpqxq\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.217115 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.216182    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.375264 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.375035    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:09.773967 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.770041    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"]
Oct 29 21:55:09.780719 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:09.780686    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddab3d72_73bf_4033_be01_6177dddff66a.slice/crio-2dc598583b9ecc726b42d1a697bd0fd733dfad52d85c8519b0b97c35e6701262 WatchSource:0}: Error finding container 2dc598583b9ecc726b42d1a697bd0fd733dfad52d85c8519b0b97c35e6701262: Status 404 returned error can't find the container with id 2dc598583b9ecc726b42d1a697bd0fd733dfad52d85c8519b0b97c35e6701262
Oct 29 21:55:09.801444 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.800103    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:09.801444 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.800196    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:09.801444 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.800235    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:09.911077 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:09.911041    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:55:10.597916 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:10.597881    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" event={"ID":"ddab3d72-73bf-4033-be01-6177dddff66a","Type":"ContainerStarted","Data":"49b3ee64fd31e713c9cdb45735adec0534d19db33d957d3c3794824a8536c8a5"}
Oct 29 21:55:10.599390 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:10.599360    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" event={"ID":"ddab3d72-73bf-4033-be01-6177dddff66a","Type":"ContainerStarted","Data":"2dc598583b9ecc726b42d1a697bd0fd733dfad52d85c8519b0b97c35e6701262"}
Oct 29 21:55:10.663651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:10.660813    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" podStartSLOduration=2.660790644 podStartE2EDuration="2.660790644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:10.659732877 +0000 UTC m=+483.159775230" watchObservedRunningTime="2026-10-29 21:55:10.660790644 +0000 UTC m=+483.160832944"
Oct 29 21:55:11.376102 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.376011    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:11.383455 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.383427    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:11.609643 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.609593    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g" event={"ID":"9e80dce6-46ef-46c9-84cd-677cd9e618f0","Type":"ContainerStarted","Data":"0586736d83b0044ed423a5169bc57e99081396046408e0cc2cafa027c680d127"}
Oct 29 21:55:11.610071 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.609724    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"
Oct 29 21:55:11.615040 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.614926    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx" event={"ID":"12f60c90-fa0c-4d19-9e5f-42c768a911f4","Type":"ContainerStarted","Data":"9cbe1943bd64ec1dc7b3a9d225452fc316b608d75f190488c16d5c5170ceaaae"}
Oct 29 21:55:11.615040 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.615009    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"
Oct 29 21:55:11.618279 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.618199    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct" event={"ID":"efd56092-21a8-45fa-9cac-9a6094fc5a50","Type":"ContainerStarted","Data":"7485a4dfd6d05ada65551d7139ef89aeb8e6c108c5fa508b0d0d070548e5b5fa"}
Oct 29 21:55:11.618279 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.618250    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"
Oct 29 21:55:11.619988 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.619961    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz" event={"ID":"89951ee0-b7e8-48dd-8441-4da86c0fbc62","Type":"ContainerStarted","Data":"3239ecce0640c66f8d933b2b25285766b41bb5708b2d44d72295570b0d689fe7"}
Oct 29 21:55:11.620103 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.620015    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:11.620285 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.620267    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"
Oct 29 21:55:11.621149 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.621128    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:11.655570 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.655512    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g" podStartSLOduration=1.800518547 podStartE2EDuration="4.655495095s" totalImagesPullingTime="2.854976548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:11.653395287 +0000 UTC m=+484.153437587" watchObservedRunningTime="2026-10-29 21:55:11.655495095 +0000 UTC m=+484.155537394"
Oct 29 21:55:11.694610 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.694566    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct" podStartSLOduration=1.579453521 podStartE2EDuration="5.694552594s" totalImagesPullingTime="4.115099073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:11.693821898 +0000 UTC m=+484.193864197" watchObservedRunningTime="2026-10-29 21:55:11.694552594 +0000 UTC m=+484.194594892"
Oct 29 21:55:11.735756 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.735708    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx" podStartSLOduration=2.054929156 podStartE2EDuration="4.735694453s" totalImagesPullingTime="2.680765297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:11.734369984 +0000 UTC m=+484.234412285" watchObservedRunningTime="2026-10-29 21:55:11.735694453 +0000 UTC m=+484.235736751"
Oct 29 21:55:11.828737 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:11.828688    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz" podStartSLOduration=2.402420531 podStartE2EDuration="6.828671322s" totalImagesPullingTime="4.426250791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:11.826326123 +0000 UTC m=+484.326368435" watchObservedRunningTime="2026-10-29 21:55:11.828671322 +0000 UTC m=+484.328713621"
Oct 29 21:55:12.630108 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:12.630068    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5" event={"ID":"97afdabd-be6a-4a98-ad49-77909430ed98","Type":"ContainerStarted","Data":"058b442e8a4f4948f8cbd2e9dfa8cf6b5f2594fc2419fc71305cd8b7c35ea910"}
Oct 29 21:55:12.630570 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:12.630515    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"
Oct 29 21:55:12.676058 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:12.676003    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5" podStartSLOduration=2.522791005 podStartE2EDuration="6.675989684s" totalImagesPullingTime="4.153198679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:12.673271747 +0000 UTC m=+485.173314046" watchObservedRunningTime="2026-10-29 21:55:12.675989684 +0000 UTC m=+485.176031982"
Oct 29 21:55:16.673559 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:16.673528    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:55:18.184586 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.184552    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"]
Oct 29 21:55:18.228647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.227841    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"]
Oct 29 21:55:18.228647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.228001    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"
Oct 29 21:55:18.261858 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.261822    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a3c02e\""
Oct 29 21:55:18.262260 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.262204    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a3c02e-dockercfg-x9frw\""
Oct 29 21:55:18.282682 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.271415    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"]
Oct 29 21:55:18.350873 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.350714    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f63896-629a-4327-a138-279748c035bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q\" (UID: \"86f63896-629a-4327-a138-279748c035bd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"
Oct 29 21:55:18.351064 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.351028    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdlc\" (UniqueName: \"kubernetes.io/projected/86f63896-629a-4327-a138-279748c035bd-kube-api-access-rxdlc\") pod \"payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q\" (UID: \"86f63896-629a-4327-a138-279748c035bd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"
Oct 29 21:55:18.373373 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.373316    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"]
Oct 29 21:55:18.373570 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.373486    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"
Oct 29 21:55:18.379163 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.379135    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"]
Oct 29 21:55:18.402839 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.385237    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"]
Oct 29 21:55:18.402839 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.385351    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"
Oct 29 21:55:18.413648 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.413604    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 21:55:18.424613 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.419619    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 21:55:18.424932 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.424916    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 21:55:18.426130 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.426115    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 21:55:18.452334 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.452275    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxsjb\" (UniqueName: \"kubernetes.io/projected/7b205272-e4c0-432c-860d-162d9e80f950-kube-api-access-jxsjb\") pod \"payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb\" (UID: \"7b205272-e4c0-432c-860d-162d9e80f950\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"
Oct 29 21:55:18.452569 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.452550    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6\" (UID: \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"
Oct 29 21:55:18.452744 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.452726    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f63896-629a-4327-a138-279748c035bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q\" (UID: \"86f63896-629a-4327-a138-279748c035bd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"
Oct 29 21:55:18.452811 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.452794    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2dnl\" (UniqueName: \"kubernetes.io/projected/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-kube-api-access-c2dnl\") pod \"maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6\" (UID: \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"
Oct 29 21:55:18.452864 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.452844    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdlc\" (UniqueName: \"kubernetes.io/projected/86f63896-629a-4327-a138-279748c035bd-kube-api-access-rxdlc\") pod \"payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q\" (UID: \"86f63896-629a-4327-a138-279748c035bd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"
Oct 29 21:55:18.452930 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.452887    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b205272-e4c0-432c-860d-162d9e80f950-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb\" (UID: \"7b205272-e4c0-432c-860d-162d9e80f950\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"
Oct 29 21:55:18.452980 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.452936    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6\" (UID: \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"
Oct 29 21:55:18.453562 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.453540    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f63896-629a-4327-a138-279748c035bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q\" (UID: \"86f63896-629a-4327-a138-279748c035bd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"
Oct 29 21:55:18.476470 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.476444    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdlc\" (UniqueName: \"kubernetes.io/projected/86f63896-629a-4327-a138-279748c035bd-kube-api-access-rxdlc\") pod \"payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q\" (UID: \"86f63896-629a-4327-a138-279748c035bd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"
Oct 29 21:55:18.553878 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.553837    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"
Oct 29 21:55:18.554220 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.554196    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2dnl\" (UniqueName: \"kubernetes.io/projected/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-kube-api-access-c2dnl\") pod \"maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6\" (UID: \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"
Oct 29 21:55:18.554311 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.554288    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b205272-e4c0-432c-860d-162d9e80f950-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb\" (UID: \"7b205272-e4c0-432c-860d-162d9e80f950\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"
Oct 29 21:55:18.554386 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.554354    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6\" (UID: \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"
Oct 29 21:55:18.554449 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.554422    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxsjb\" (UniqueName: \"kubernetes.io/projected/7b205272-e4c0-432c-860d-162d9e80f950-kube-api-access-jxsjb\") pod \"payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb\" (UID: \"7b205272-e4c0-432c-860d-162d9e80f950\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"
Oct 29 21:55:18.554583 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.554540    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6\" (UID: \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"
Oct 29 21:55:18.555037 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.554982    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b205272-e4c0-432c-860d-162d9e80f950-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb\" (UID: \"7b205272-e4c0-432c-860d-162d9e80f950\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"
Oct 29 21:55:18.555037 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.555014    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6\" (UID: \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"
Oct 29 21:55:18.557172 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.557153    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6\" (UID: \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"
Oct 29 21:55:18.577713 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.577688    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxsjb\" (UniqueName: \"kubernetes.io/projected/7b205272-e4c0-432c-860d-162d9e80f950-kube-api-access-jxsjb\") pod \"payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb\" (UID: \"7b205272-e4c0-432c-860d-162d9e80f950\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"
Oct 29 21:55:18.577890 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.577868    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2dnl\" (UniqueName: \"kubernetes.io/projected/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-kube-api-access-c2dnl\") pod \"maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6\" (UID: \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"
Oct 29 21:55:18.728080 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.727135    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"
Oct 29 21:55:18.738392 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.738364    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"
Oct 29 21:55:18.940652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:18.938028    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"]
Oct 29 21:55:18.946357 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:18.946318    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f63896_629a_4327_a138_279748c035bd.slice/crio-8f25816e7a98a60194f6eab1f6ffe580627a0de32b9df76bffab89ddb76ca85b WatchSource:0}: Error finding container 8f25816e7a98a60194f6eab1f6ffe580627a0de32b9df76bffab89ddb76ca85b: Status 404 returned error can't find the container with id 8f25816e7a98a60194f6eab1f6ffe580627a0de32b9df76bffab89ddb76ca85b
Oct 29 21:55:19.083803 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:19.082620    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"]
Oct 29 21:55:19.164855 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:19.164832    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"]
Oct 29 21:55:19.167573 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:19.167491    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3ea07b_15c1_4ba9_b1ba_bfa8f3c279a6.slice/crio-5eb90b8538526ed3339426e9ffd68886bfaf89c1afe8fddc03aa01170faddd5b WatchSource:0}: Error finding container 5eb90b8538526ed3339426e9ffd68886bfaf89c1afe8fddc03aa01170faddd5b: Status 404 returned error can't find the container with id 5eb90b8538526ed3339426e9ffd68886bfaf89c1afe8fddc03aa01170faddd5b
Oct 29 21:55:19.680958 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:19.680917    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6" event={"ID":"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6","Type":"ContainerStarted","Data":"5eb90b8538526ed3339426e9ffd68886bfaf89c1afe8fddc03aa01170faddd5b"}
Oct 29 21:55:19.682532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:19.682505    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q" event={"ID":"86f63896-629a-4327-a138-279748c035bd","Type":"ContainerStarted","Data":"89a643d8590a7124072f57fb3793e97014460795b3ed289c8f0172a844ffcfc5"}
Oct 29 21:55:19.682656 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:19.682557    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q" event={"ID":"86f63896-629a-4327-a138-279748c035bd","Type":"ContainerStarted","Data":"8f25816e7a98a60194f6eab1f6ffe580627a0de32b9df76bffab89ddb76ca85b"}
Oct 29 21:55:19.683453 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:19.682856    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"
Oct 29 21:55:19.686514 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:19.685798    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb" event={"ID":"7b205272-e4c0-432c-860d-162d9e80f950","Type":"ContainerStarted","Data":"f7cc9d56f53ee9826288e4c70970cd7d3f9f94fec4d1b0011c1aa09bf88f511c"}
Oct 29 21:55:19.686514 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:19.685852    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb" event={"ID":"7b205272-e4c0-432c-860d-162d9e80f950","Type":"ContainerStarted","Data":"f53ee408e0caa4b804d12935b010e527a23ecca64d086b1aa722114b79bec96b"}
Oct 29 21:55:19.686514 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:19.686454    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"
Oct 29 21:55:19.738654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:19.732865    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q" podStartSLOduration=1.7328435629999999 podStartE2EDuration="1.732843563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:19.731055927 +0000 UTC m=+492.231098228" watchObservedRunningTime="2026-10-29 21:55:19.732843563 +0000 UTC m=+492.232885864"
Oct 29 21:55:19.773862 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:19.773811    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb" podStartSLOduration=1.7737967590000001 podStartE2EDuration="1.773796759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:19.772778892 +0000 UTC m=+492.272821190" watchObservedRunningTime="2026-10-29 21:55:19.773796759 +0000 UTC m=+492.273839058"
Oct 29 21:55:22.251655 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.251007    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"]
Oct 29 21:55:22.256543 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.256526    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"
Oct 29 21:55:22.278127 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.278103    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"]
Oct 29 21:55:22.367603 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.367564    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrvsr\" (UniqueName: \"kubernetes.io/projected/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-kube-api-access-zrvsr\") pod \"maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9\" (UID: \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"
Oct 29 21:55:22.367779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.367659    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9\" (UID: \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"
Oct 29 21:55:22.367779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.367752    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9\" (UID: \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"
Oct 29 21:55:22.469052 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.469018    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9\" (UID: \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"
Oct 29 21:55:22.469220 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.469141    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrvsr\" (UniqueName: \"kubernetes.io/projected/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-kube-api-access-zrvsr\") pod \"maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9\" (UID: \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"
Oct 29 21:55:22.469220 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.469196    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9\" (UID: \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"
Oct 29 21:55:22.469820 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.469797    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9\" (UID: \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"
Oct 29 21:55:22.472727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.472674    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9\" (UID: \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"
Oct 29 21:55:22.527928 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.527847    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrvsr\" (UniqueName: \"kubernetes.io/projected/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-kube-api-access-zrvsr\") pod \"maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9\" (UID: \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"
Oct 29 21:55:22.576797 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.576766    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"
Oct 29 21:55:22.645683 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.645651    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"
Oct 29 21:55:22.645865 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.645836    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"
Oct 29 21:55:22.647585 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:22.647562    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"
Oct 29 21:55:23.020582 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:23.020558    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"]
Oct 29 21:55:23.024592 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:23.024566    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8cf832_cda9_49f0_b174_dd2b0c0cbade.slice/crio-6de14b8ef0ec622ba5f4aaf628c36b133caeb9d8b55c92cbc8bceb3c4b5b3046 WatchSource:0}: Error finding container 6de14b8ef0ec622ba5f4aaf628c36b133caeb9d8b55c92cbc8bceb3c4b5b3046: Status 404 returned error can't find the container with id 6de14b8ef0ec622ba5f4aaf628c36b133caeb9d8b55c92cbc8bceb3c4b5b3046
Oct 29 21:55:23.639601 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:23.639571    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"
Oct 29 21:55:23.775114 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:23.775076    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9" event={"ID":"1e8cf832-cda9-49f0-b174-dd2b0c0cbade","Type":"ContainerStarted","Data":"6de14b8ef0ec622ba5f4aaf628c36b133caeb9d8b55c92cbc8bceb3c4b5b3046"}
Oct 29 21:55:24.647817 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:24.646203    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"
Oct 29 21:55:24.792891 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:24.792800    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6" event={"ID":"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6","Type":"ContainerStarted","Data":"6a9213b9e712955819793aab965dcf8c7743133ddf10d1e8f9f5934c91d51165"}
Oct 29 21:55:24.839678 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:24.839582    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6" podStartSLOduration=1.970945604 podStartE2EDuration="6.839562649s" totalImagesPullingTime="4.868617045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:24.835894255 +0000 UTC m=+497.335936555" watchObservedRunningTime="2026-10-29 21:55:24.839562649 +0000 UTC m=+497.339604950"
Oct 29 21:55:25.802097 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:25.802053    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9" event={"ID":"1e8cf832-cda9-49f0-b174-dd2b0c0cbade","Type":"ContainerStarted","Data":"7e825d7b09cb5bd704757914b9b6d70ffff834ff393bf063150554d6139c2665"}
Oct 29 21:55:25.805262 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:25.805231    7082 generic.go:356] "Generic (PLEG): container finished" podID="fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6" containerID="6a9213b9e712955819793aab965dcf8c7743133ddf10d1e8f9f5934c91d51165" exitCode=0
Oct 29 21:55:25.805406 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:25.805272    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6" event={"ID":"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6","Type":"ContainerDied","Data":"6a9213b9e712955819793aab965dcf8c7743133ddf10d1e8f9f5934c91d51165"}
Oct 29 21:55:25.894886 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:25.894809    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9" podStartSLOduration=2.000478528 podStartE2EDuration="3.894791998s" totalImagesPullingTime="1.89431347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:25.848324552 +0000 UTC m=+498.348366854" watchObservedRunningTime="2026-10-29 21:55:25.894791998 +0000 UTC m=+498.394834336"
Oct 29 21:55:26.815132 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:26.815092    7082 generic.go:356] "Generic (PLEG): container finished" podID="1e8cf832-cda9-49f0-b174-dd2b0c0cbade" containerID="7e825d7b09cb5bd704757914b9b6d70ffff834ff393bf063150554d6139c2665" exitCode=0
Oct 29 21:55:26.815620 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:26.815179    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9" event={"ID":"1e8cf832-cda9-49f0-b174-dd2b0c0cbade","Type":"ContainerDied","Data":"7e825d7b09cb5bd704757914b9b6d70ffff834ff393bf063150554d6139c2665"}
Oct 29 21:55:27.001578 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.001551    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"
Oct 29 21:55:27.073324 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.073226    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-openshift-ca\") pod \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\" (UID: \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\") "
Oct 29 21:55:27.073502 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.073341    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-xks-ca\") pod \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\" (UID: \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\") "
Oct 29 21:55:27.073502 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.073387    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-kube-api-access-c2dnl\" (UniqueName: \"kubernetes.io/projected/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-kube-api-access-c2dnl\") pod \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\" (UID: \"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6\") "
Oct 29 21:55:27.074653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.074597    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-openshift-ca" pod "fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6" (UID: "fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:27.077269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.077242    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-xks-ca" pod "fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6" (UID: "fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:27.087573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.087516    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-kube-api-access-c2dnl" pod "fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6" (UID: "fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6"). InnerVolumeSpecName "kube-api-access-c2dnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:27.175429 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.175389    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:27.176381 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.175432    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:27.176381 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.175451    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2dnl\" (UniqueName: \"kubernetes.io/projected/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6-kube-api-access-c2dnl\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:27.833044 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.832997    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6" event={"ID":"fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6","Type":"ContainerDied","Data":"5eb90b8538526ed3339426e9ffd68886bfaf89c1afe8fddc03aa01170faddd5b"}
Oct 29 21:55:27.833617 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.833070    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eb90b8538526ed3339426e9ffd68886bfaf89c1afe8fddc03aa01170faddd5b"
Oct 29 21:55:27.833617 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.833141    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"
Oct 29 21:55:27.836649 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.836585    7082 generic.go:356] "Generic (PLEG): container finished" podID="05954176-b158-4666-933b-334e368ef303" containerID="c9d21e6d3dde9c1560cd50b67969b3838a9b808d9b518f976cc6382e5779bd90" exitCode=0
Oct 29 21:55:27.836841 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.836815    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" event={"ID":"05954176-b158-4666-933b-334e368ef303","Type":"ContainerDied","Data":"c9d21e6d3dde9c1560cd50b67969b3838a9b808d9b518f976cc6382e5779bd90"}
Oct 29 21:55:27.838024 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.837976    7082 scope.go:122] "RemoveContainer" containerID="c9d21e6d3dde9c1560cd50b67969b3838a9b808d9b518f976cc6382e5779bd90"
Oct 29 21:55:27.840295 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.840257    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:27.840416 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.840351    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:27.840416 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:27.840405    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:28.081073 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.081044    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"
Oct 29 21:55:28.192257 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.191310    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-kube-api-access-zrvsr\" (UniqueName: \"kubernetes.io/projected/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-kube-api-access-zrvsr\") pod \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\" (UID: \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\") "
Oct 29 21:55:28.192257 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.191442    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-openshift-ca\") pod \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\" (UID: \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\") "
Oct 29 21:55:28.192257 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.191490    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-xks-ca\") pod \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\" (UID: \"1e8cf832-cda9-49f0-b174-dd2b0c0cbade\") "
Oct 29 21:55:28.194290 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.194249    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-openshift-ca" pod "1e8cf832-cda9-49f0-b174-dd2b0c0cbade" (UID: "1e8cf832-cda9-49f0-b174-dd2b0c0cbade"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:28.207154 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.201434    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-xks-ca" pod "1e8cf832-cda9-49f0-b174-dd2b0c0cbade" (UID: "1e8cf832-cda9-49f0-b174-dd2b0c0cbade"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:28.218683 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.215882    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-kube-api-access-zrvsr" pod "1e8cf832-cda9-49f0-b174-dd2b0c0cbade" (UID: "1e8cf832-cda9-49f0-b174-dd2b0c0cbade"). InnerVolumeSpecName "kube-api-access-zrvsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:28.292945 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.292907    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:28.292945 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.292942    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:28.293149 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.292957    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrvsr\" (UniqueName: \"kubernetes.io/projected/1e8cf832-cda9-49f0-b174-dd2b0c0cbade-kube-api-access-zrvsr\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:28.378128 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.378092    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:55:28.389748 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.389714    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"]
Oct 29 21:55:28.390297 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.390278    7082 cpu_manager.go:414] "containerMap: removing container" podUID="1e8cf832-cda9-49f0-b174-dd2b0c0cbade" containerName="revoke-keys"
Oct 29 21:55:28.390297 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.390296    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8cf832-cda9-49f0-b174-dd2b0c0cbade" containerName="revoke-keys"
Oct 29 21:55:28.390465 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.390308    7082 cpu_manager.go:414] "containerMap: removing container" podUID="fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6" containerName="revoke-keys"
Oct 29 21:55:28.390465 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.390313    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6" containerName="revoke-keys"
Oct 29 21:55:28.390465 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.390394    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6" containerName="revoke-keys"
Oct 29 21:55:28.390465 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.390402    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8cf832-cda9-49f0-b174-dd2b0c0cbade" containerName="revoke-keys"
Oct 29 21:55:28.393389 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.393365    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"
Oct 29 21:55:28.468276 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.468153    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"]
Oct 29 21:55:28.496461 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.496406    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-de7186ef-g479m\" (UID: \"895a41f7-ce04-428c-9ba8-0afc0823aa88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"
Oct 29 21:55:28.496698 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.496599    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-de7186ef-g479m\" (UID: \"895a41f7-ce04-428c-9ba8-0afc0823aa88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"
Oct 29 21:55:28.496788 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.496757    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bblf\" (UniqueName: \"kubernetes.io/projected/895a41f7-ce04-428c-9ba8-0afc0823aa88-kube-api-access-6bblf\") pod \"maas-api-revoke-keys-e2e-ait-de7186ef-g479m\" (UID: \"895a41f7-ce04-428c-9ba8-0afc0823aa88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"
Oct 29 21:55:28.527686 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:55:28.527433    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:55:28.572707 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.572669    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"]
Oct 29 21:55:28.577165 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.577143    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"
Oct 29 21:55:28.617702 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.600532    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-de7186ef-g479m\" (UID: \"895a41f7-ce04-428c-9ba8-0afc0823aa88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"
Oct 29 21:55:28.617702 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.600689    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-de7186ef-g479m\" (UID: \"895a41f7-ce04-428c-9ba8-0afc0823aa88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"
Oct 29 21:55:28.617702 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.600776    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bblf\" (UniqueName: \"kubernetes.io/projected/895a41f7-ce04-428c-9ba8-0afc0823aa88-kube-api-access-6bblf\") pod \"maas-api-revoke-keys-e2e-ait-de7186ef-g479m\" (UID: \"895a41f7-ce04-428c-9ba8-0afc0823aa88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"
Oct 29 21:55:28.617702 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.601976    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-de7186ef-g479m\" (UID: \"895a41f7-ce04-428c-9ba8-0afc0823aa88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"
Oct 29 21:55:28.645181 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.645141    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"]
Oct 29 21:55:28.652971 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.649620    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-de7186ef-g479m\" (UID: \"895a41f7-ce04-428c-9ba8-0afc0823aa88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"
Oct 29 21:55:28.653178 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.653153    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bblf\" (UniqueName: \"kubernetes.io/projected/895a41f7-ce04-428c-9ba8-0afc0823aa88-kube-api-access-6bblf\") pod \"maas-api-revoke-keys-e2e-ait-de7186ef-g479m\" (UID: \"895a41f7-ce04-428c-9ba8-0afc0823aa88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"
Oct 29 21:55:28.703463 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.703120    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af7a0574-17ca-4088-9470-d151a6177705-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x\" (UID: \"af7a0574-17ca-4088-9470-d151a6177705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"
Oct 29 21:55:28.703463 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.703194    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af7a0574-17ca-4088-9470-d151a6177705-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x\" (UID: \"af7a0574-17ca-4088-9470-d151a6177705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"
Oct 29 21:55:28.703463 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.703324    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k75tq\" (UniqueName: \"kubernetes.io/projected/af7a0574-17ca-4088-9470-d151a6177705-kube-api-access-k75tq\") pod \"maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x\" (UID: \"af7a0574-17ca-4088-9470-d151a6177705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"
Oct 29 21:55:28.760891 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.746861    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"
Oct 29 21:55:28.824687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.808398    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k75tq\" (UniqueName: \"kubernetes.io/projected/af7a0574-17ca-4088-9470-d151a6177705-kube-api-access-k75tq\") pod \"maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x\" (UID: \"af7a0574-17ca-4088-9470-d151a6177705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"
Oct 29 21:55:28.824687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.808560    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af7a0574-17ca-4088-9470-d151a6177705-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x\" (UID: \"af7a0574-17ca-4088-9470-d151a6177705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"
Oct 29 21:55:28.824687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.808618    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af7a0574-17ca-4088-9470-d151a6177705-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x\" (UID: \"af7a0574-17ca-4088-9470-d151a6177705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"
Oct 29 21:55:28.824687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.809542    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af7a0574-17ca-4088-9470-d151a6177705-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x\" (UID: \"af7a0574-17ca-4088-9470-d151a6177705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"
Oct 29 21:55:28.832797 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.832734    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af7a0574-17ca-4088-9470-d151a6177705-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x\" (UID: \"af7a0574-17ca-4088-9470-d151a6177705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"
Oct 29 21:55:28.847605 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.847584    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"
Oct 29 21:55:28.852937 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.847597    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9" event={"ID":"1e8cf832-cda9-49f0-b174-dd2b0c0cbade","Type":"ContainerDied","Data":"6de14b8ef0ec622ba5f4aaf628c36b133caeb9d8b55c92cbc8bceb3c4b5b3046"}
Oct 29 21:55:28.852937 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.848672    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6de14b8ef0ec622ba5f4aaf628c36b133caeb9d8b55c92cbc8bceb3c4b5b3046"
Oct 29 21:55:28.852937 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.850154    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" event={"ID":"05954176-b158-4666-933b-334e368ef303","Type":"ContainerStarted","Data":"e2352975cddf63b67507a6d3411d0c1e3884dfadb19450286cf2696e664b9c65"}
Oct 29 21:55:28.887090 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.858129    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k75tq\" (UniqueName: \"kubernetes.io/projected/af7a0574-17ca-4088-9470-d151a6177705-kube-api-access-k75tq\") pod \"maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x\" (UID: \"af7a0574-17ca-4088-9470-d151a6177705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"
Oct 29 21:55:28.961583 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:28.961063    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"
Oct 29 21:55:29.274966 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.274878    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"]
Oct 29 21:55:29.345108 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.345081    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"]
Oct 29 21:55:29.347769 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:29.347537    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7a0574_17ca_4088_9470_d151a6177705.slice/crio-5900a6efc5f2fbaf271a93b6da74d45528b0e272627e3212fd72431a746006a9 WatchSource:0}: Error finding container 5900a6efc5f2fbaf271a93b6da74d45528b0e272627e3212fd72431a746006a9: Status 404 returned error can't find the container with id 5900a6efc5f2fbaf271a93b6da74d45528b0e272627e3212fd72431a746006a9
Oct 29 21:55:29.447928 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.447464    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:55:29.448334 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.448298    7082 patch_prober.go:29] interesting pod/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 29 21:55:29.448485 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.448351    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 29 21:55:29.862007 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.861804    7082 generic.go:356] "Generic (PLEG): container finished" podID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerID="5d9403b6be3b05c4cff055788b4b413ba89d028bed746bd7426e56322f24983f" exitCode=0
Oct 29 21:55:29.862007 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.861962    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" event={"ID":"45bb9e0c-17cf-41f8-b79e-841fa270c02b","Type":"ContainerDied","Data":"5d9403b6be3b05c4cff055788b4b413ba89d028bed746bd7426e56322f24983f"}
Oct 29 21:55:29.862644 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.862452    7082 scope.go:122] "RemoveContainer" containerID="5d9403b6be3b05c4cff055788b4b413ba89d028bed746bd7426e56322f24983f"
Oct 29 21:55:29.865439 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.865399    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:29.865549 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.865493    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:29.865549 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.865539    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:29.870202 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.870177    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m" event={"ID":"895a41f7-ce04-428c-9ba8-0afc0823aa88","Type":"ContainerStarted","Data":"e606676abf49bf687dc5e6fa2cf1621939a2624e59abb86401e32b9200445748"}
Oct 29 21:55:29.870408 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.870372    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m" event={"ID":"895a41f7-ce04-428c-9ba8-0afc0823aa88","Type":"ContainerStarted","Data":"8c79694ef8cff984fa12ca3aec5329755b481c561d1b5b02b1958f68cd8f3c88"}
Oct 29 21:55:29.872189 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.872118    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x" event={"ID":"af7a0574-17ca-4088-9470-d151a6177705","Type":"ContainerStarted","Data":"5900a6efc5f2fbaf271a93b6da74d45528b0e272627e3212fd72431a746006a9"}
Oct 29 21:55:29.951107 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:29.951042    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m" podStartSLOduration=1.951013964 podStartE2EDuration="1.951013964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:29.950090064 +0000 UTC m=+502.450132354" watchObservedRunningTime="2026-10-29 21:55:29.951013964 +0000 UTC m=+502.451056253"
Oct 29 21:55:30.449094 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:30.449054    7082 patch_prober.go:29] interesting pod/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 29 21:55:30.449281 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:30.449126    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 29 21:55:30.771851 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:30.771764    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"
Oct 29 21:55:30.946672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:30.945751    7082 generic.go:356] "Generic (PLEG): container finished" podID="af7a0574-17ca-4088-9470-d151a6177705" containerID="74780f655d34e98403e818b1c8cb3d486dcd9de3283c7638dc5bc0b01b3039c4" exitCode=0
Oct 29 21:55:30.946672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:30.945991    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x" event={"ID":"af7a0574-17ca-4088-9470-d151a6177705","Type":"ContainerDied","Data":"74780f655d34e98403e818b1c8cb3d486dcd9de3283c7638dc5bc0b01b3039c4"}
Oct 29 21:55:31.027709 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:31.024349    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" event={"ID":"45bb9e0c-17cf-41f8-b79e-841fa270c02b","Type":"ContainerStarted","Data":"6ef82de030284440b0dc4fd91d63c1b0aff041ca6e12711453e4fa3106957132"}
Oct 29 21:55:31.064653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:31.054496    7082 generic.go:356] "Generic (PLEG): container finished" podID="895a41f7-ce04-428c-9ba8-0afc0823aa88" containerID="e606676abf49bf687dc5e6fa2cf1621939a2624e59abb86401e32b9200445748" exitCode=0
Oct 29 21:55:31.064653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:31.054577    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m" event={"ID":"895a41f7-ce04-428c-9ba8-0afc0823aa88","Type":"ContainerDied","Data":"e606676abf49bf687dc5e6fa2cf1621939a2624e59abb86401e32b9200445748"}
Oct 29 21:55:31.447652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:31.447062    7082 patch_prober.go:29] interesting pod/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 29 21:55:31.447652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:31.447141    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 29 21:55:31.504652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:31.503774    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:55:31.721706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:31.720948    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:55:31.721706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:31.721119    7082 patch_prober.go:29] interesting pod/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 29 21:55:31.721706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:31.721171    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 29 21:55:31.767651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:31.767350    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"
Oct 29 21:55:32.454097 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:32.454010    7082 patch_prober.go:29] interesting pod/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 29 21:55:32.454735 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:32.454698    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 29 21:55:32.577482 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:32.577005    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:55:32.782711 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:32.742791    7082 patch_prober.go:29] interesting pod/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 29 21:55:32.782711 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:32.742855    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 29 21:55:32.815959 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:32.815922    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"
Oct 29 21:55:33.031778 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.031739    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"]
Oct 29 21:55:33.170482 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.170335    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-xks-ca\") pod \"895a41f7-ce04-428c-9ba8-0afc0823aa88\" (UID: \"895a41f7-ce04-428c-9ba8-0afc0823aa88\") "
Oct 29 21:55:33.171332 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.171309    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895a41f7-ce04-428c-9ba8-0afc0823aa88-kube-api-access-6bblf\" (UniqueName: \"kubernetes.io/projected/895a41f7-ce04-428c-9ba8-0afc0823aa88-kube-api-access-6bblf\") pod \"895a41f7-ce04-428c-9ba8-0afc0823aa88\" (UID: \"895a41f7-ce04-428c-9ba8-0afc0823aa88\") "
Oct 29 21:55:33.171563 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.171549    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-openshift-ca\") pod \"895a41f7-ce04-428c-9ba8-0afc0823aa88\" (UID: \"895a41f7-ce04-428c-9ba8-0afc0823aa88\") "
Oct 29 21:55:33.172429 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.172407    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-openshift-ca" pod "895a41f7-ce04-428c-9ba8-0afc0823aa88" (UID: "895a41f7-ce04-428c-9ba8-0afc0823aa88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:33.180291 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.179694    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895a41f7-ce04-428c-9ba8-0afc0823aa88-kube-api-access-6bblf" pod "895a41f7-ce04-428c-9ba8-0afc0823aa88" (UID: "895a41f7-ce04-428c-9ba8-0afc0823aa88"). InnerVolumeSpecName "kube-api-access-6bblf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:33.180291 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.179858    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"
Oct 29 21:55:33.180291 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.180230    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-xks-ca" pod "895a41f7-ce04-428c-9ba8-0afc0823aa88" (UID: "895a41f7-ce04-428c-9ba8-0afc0823aa88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:33.291662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.284514    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af7a0574-17ca-4088-9470-d151a6177705-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af7a0574-17ca-4088-9470-d151a6177705-maas-api-xks-ca\") pod \"af7a0574-17ca-4088-9470-d151a6177705\" (UID: \"af7a0574-17ca-4088-9470-d151a6177705\") "
Oct 29 21:55:33.291662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.284565    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af7a0574-17ca-4088-9470-d151a6177705-kube-api-access-k75tq\" (UniqueName: \"kubernetes.io/projected/af7a0574-17ca-4088-9470-d151a6177705-kube-api-access-k75tq\") pod \"af7a0574-17ca-4088-9470-d151a6177705\" (UID: \"af7a0574-17ca-4088-9470-d151a6177705\") "
Oct 29 21:55:33.291662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.284748    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af7a0574-17ca-4088-9470-d151a6177705-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af7a0574-17ca-4088-9470-d151a6177705-maas-api-openshift-ca\") pod \"af7a0574-17ca-4088-9470-d151a6177705\" (UID: \"af7a0574-17ca-4088-9470-d151a6177705\") "
Oct 29 21:55:33.291662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.285176    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bblf\" (UniqueName: \"kubernetes.io/projected/895a41f7-ce04-428c-9ba8-0afc0823aa88-kube-api-access-6bblf\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:33.291662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.285192    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:33.291662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.285210    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895a41f7-ce04-428c-9ba8-0afc0823aa88-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:33.356668 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.355888    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7a0574-17ca-4088-9470-d151a6177705-maas-api-openshift-ca" pod "af7a0574-17ca-4088-9470-d151a6177705" (UID: "af7a0574-17ca-4088-9470-d151a6177705"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:33.382930 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.382891    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x" event={"ID":"af7a0574-17ca-4088-9470-d151a6177705","Type":"ContainerDied","Data":"5900a6efc5f2fbaf271a93b6da74d45528b0e272627e3212fd72431a746006a9"}
Oct 29 21:55:33.383463 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.383262    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5900a6efc5f2fbaf271a93b6da74d45528b0e272627e3212fd72431a746006a9"
Oct 29 21:55:33.383463 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.382990    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"
Oct 29 21:55:33.419659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.389125    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"
Oct 29 21:55:33.419659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.393703    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m" event={"ID":"895a41f7-ce04-428c-9ba8-0afc0823aa88","Type":"ContainerDied","Data":"8c79694ef8cff984fa12ca3aec5329755b481c561d1b5b02b1958f68cd8f3c88"}
Oct 29 21:55:33.419659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.393777    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c79694ef8cff984fa12ca3aec5329755b481c561d1b5b02b1958f68cd8f3c88"
Oct 29 21:55:33.452710 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.427124    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af7a0574-17ca-4088-9470-d151a6177705-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:33.452710 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.451996    7082 patch_prober.go:29] interesting pod/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 29 21:55:33.452710 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.452109    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 29 21:55:33.452987 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.452787    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7a0574-17ca-4088-9470-d151a6177705-kube-api-access-k75tq" pod "af7a0574-17ca-4088-9470-d151a6177705" (UID: "af7a0574-17ca-4088-9470-d151a6177705"). InnerVolumeSpecName "kube-api-access-k75tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:33.624655 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.573521    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k75tq\" (UniqueName: \"kubernetes.io/projected/af7a0574-17ca-4088-9470-d151a6177705-kube-api-access-k75tq\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:33.775194 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.603802    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7a0574-17ca-4088-9470-d151a6177705-maas-api-xks-ca" pod "af7a0574-17ca-4088-9470-d151a6177705" (UID: "af7a0574-17ca-4088-9470-d151a6177705"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:33.775425 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.763265    7082 patch_prober.go:29] interesting pod/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 29 21:55:33.775583 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.775554    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 29 21:55:33.848931 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.848898    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"]
Oct 29 21:55:33.873679 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.849736    7082 cpu_manager.go:414] "containerMap: removing container" podUID="895a41f7-ce04-428c-9ba8-0afc0823aa88" containerName="revoke-keys"
Oct 29 21:55:33.873679 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.849760    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895a41f7-ce04-428c-9ba8-0afc0823aa88" containerName="revoke-keys"
Oct 29 21:55:33.873679 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.849791    7082 cpu_manager.go:414] "containerMap: removing container" podUID="af7a0574-17ca-4088-9470-d151a6177705" containerName="revoke-keys"
Oct 29 21:55:33.873679 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.849800    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7a0574-17ca-4088-9470-d151a6177705" containerName="revoke-keys"
Oct 29 21:55:33.873679 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.849969    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="895a41f7-ce04-428c-9ba8-0afc0823aa88" containerName="revoke-keys"
Oct 29 21:55:33.873679 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.849986    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7a0574-17ca-4088-9470-d151a6177705" containerName="revoke-keys"
Oct 29 21:55:33.878521 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.876297    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af7a0574-17ca-4088-9470-d151a6177705-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:33.884405 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.884377    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"]
Oct 29 21:55:33.884537 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.884488    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"
Oct 29 21:55:33.979023 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.978987    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr\" (UID: \"20b78882-caed-4e29-939b-af1aea73c25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"
Oct 29 21:55:33.979223 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.979209    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr\" (UID: \"20b78882-caed-4e29-939b-af1aea73c25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"
Oct 29 21:55:33.979346 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:33.979335    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndrcw\" (UniqueName: \"kubernetes.io/projected/20b78882-caed-4e29-939b-af1aea73c25e-kube-api-access-ndrcw\") pod \"maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr\" (UID: \"20b78882-caed-4e29-939b-af1aea73c25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"
Oct 29 21:55:34.080089 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:34.080017    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrcw\" (UniqueName: \"kubernetes.io/projected/20b78882-caed-4e29-939b-af1aea73c25e-kube-api-access-ndrcw\") pod \"maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr\" (UID: \"20b78882-caed-4e29-939b-af1aea73c25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"
Oct 29 21:55:34.080283 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:34.080155    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr\" (UID: \"20b78882-caed-4e29-939b-af1aea73c25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"
Oct 29 21:55:34.080283 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:34.080254    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr\" (UID: \"20b78882-caed-4e29-939b-af1aea73c25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"
Oct 29 21:55:34.080856 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:34.080831    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr\" (UID: \"20b78882-caed-4e29-939b-af1aea73c25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"
Oct 29 21:55:34.142659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:34.135671    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr\" (UID: \"20b78882-caed-4e29-939b-af1aea73c25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"
Oct 29 21:55:34.143305 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:34.143241    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrcw\" (UniqueName: \"kubernetes.io/projected/20b78882-caed-4e29-939b-af1aea73c25e-kube-api-access-ndrcw\") pod \"maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr\" (UID: \"20b78882-caed-4e29-939b-af1aea73c25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"
Oct 29 21:55:34.330873 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:34.330831    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"
Oct 29 21:55:34.456673 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:34.456051    7082 patch_prober.go:29] interesting pod/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 29 21:55:34.456673 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:34.456111    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 29 21:55:34.720951 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:34.720411    7082 patch_prober.go:29] interesting pod/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 29 21:55:34.720951 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:34.720486    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 29 21:55:34.748596 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:34.748543    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"]
Oct 29 21:55:34.763946 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:34.763923    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b78882_caed_4e29_939b_af1aea73c25e.slice/crio-df8912961960a0ce7dc9a8669b926b211f9e6ad4b2721d2caab024152fd235b3 WatchSource:0}: Error finding container df8912961960a0ce7dc9a8669b926b211f9e6ad4b2721d2caab024152fd235b3: Status 404 returned error can't find the container with id df8912961960a0ce7dc9a8669b926b211f9e6ad4b2721d2caab024152fd235b3
Oct 29 21:55:35.464725 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:35.464603    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:55:35.466464 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:35.466445    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:55:35.466992 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:35.466964    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr" event={"ID":"20b78882-caed-4e29-939b-af1aea73c25e","Type":"ContainerStarted","Data":"0c93eca145c60f9b8958b134a04a5411e2d82b317f0ec680a2e40738647c2d6c"}
Oct 29 21:55:35.467067 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:35.467018    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr" event={"ID":"20b78882-caed-4e29-939b-af1aea73c25e","Type":"ContainerStarted","Data":"df8912961960a0ce7dc9a8669b926b211f9e6ad4b2721d2caab024152fd235b3"}
Oct 29 21:55:35.658913 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:35.658847    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr" podStartSLOduration=2.658824611 podStartE2EDuration="2.658824611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:35.641536063 +0000 UTC m=+508.141578389" watchObservedRunningTime="2026-10-29 21:55:35.658824611 +0000 UTC m=+508.158866910"
Oct 29 21:55:35.678264 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:35.678234    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"]
Oct 29 21:55:35.678264 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:35.678272    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-64505fc5-mj2f9"]
Oct 29 21:55:35.720128 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:35.720043    7082 patch_prober.go:29] interesting pod/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 29 21:55:35.731884 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:35.731839    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 29 21:55:35.955384 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:35.955349    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:55:36.484151 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:36.484110    7082 generic.go:356] "Generic (PLEG): container finished" podID="20b78882-caed-4e29-939b-af1aea73c25e" containerID="0c93eca145c60f9b8958b134a04a5411e2d82b317f0ec680a2e40738647c2d6c" exitCode=0
Oct 29 21:55:36.484334 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:36.484155    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr" event={"ID":"20b78882-caed-4e29-939b-af1aea73c25e","Type":"ContainerDied","Data":"0c93eca145c60f9b8958b134a04a5411e2d82b317f0ec680a2e40738647c2d6c"}
Oct 29 21:55:37.000734 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.000696    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:55:37.005560 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.005493    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:55:37.056119 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.056083    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"]
Oct 29 21:55:37.070823 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.070780    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.083320 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.083291    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"]
Oct 29 21:55:37.085769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.085747    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a940bc-openshift-default-dockercfg-cl8t8\""
Oct 29 21:55:37.128398 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.128354    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5pd9\" (UniqueName: \"kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-kube-api-access-r5pd9\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.128775 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.128747    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.128912 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.128896    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84119759-f427-4a8d-960e-8b342fca0d38-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.129070 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.129055    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.129173 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.129158    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.129285 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.129270    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.129400 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.129386    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.129586 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.129569    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.129871 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.129851    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84119759-f427-4a8d-960e-8b342fca0d38-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.231544 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.231506    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.231755 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.231561    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84119759-f427-4a8d-960e-8b342fca0d38-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.231755 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.231591    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5pd9\" (UniqueName: \"kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-kube-api-access-r5pd9\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.231755 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.231673    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.231755 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.231700    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84119759-f427-4a8d-960e-8b342fca0d38-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.232007 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.231758    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.232007 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.231785    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.232007 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.231818    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.232007 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.231854    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.232007 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.231916    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.232279 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.232119    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.232337 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.232232    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.232388 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.232345    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.232442 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.232415    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84119759-f427-4a8d-960e-8b342fca0d38-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.234487 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.234462    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.234865 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.234843    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84119759-f427-4a8d-960e-8b342fca0d38-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.255425 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.255366    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.255529 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.255474    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5pd9\" (UniqueName: \"kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-kube-api-access-r5pd9\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.427911 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.427872    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:37.640845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.640816    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e8cf832-cda9-49f0-b174-dd2b0c0cbade" path="/var/lib/kubelet/pods/1e8cf832-cda9-49f0-b174-dd2b0c0cbade/volumes"
Oct 29 21:55:37.808880 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.805548    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"
Oct 29 21:55:37.820860 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.820789    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"]
Oct 29 21:55:37.822024 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:37.821993    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84119759_f427_4a8d_960e_8b342fca0d38.slice/crio-6d8aea89d5dc134774fe3387d432b32e4145bacea1f2a038553655688099d161 WatchSource:0}: Error finding container 6d8aea89d5dc134774fe3387d432b32e4145bacea1f2a038553655688099d161: Status 404 returned error can't find the container with id 6d8aea89d5dc134774fe3387d432b32e4145bacea1f2a038553655688099d161
Oct 29 21:55:37.834732 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.834689    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:37.834848 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.834781    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:37.834848 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.834825    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:37.945454 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.945428    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b78882-caed-4e29-939b-af1aea73c25e-kube-api-access-ndrcw\" (UniqueName: \"kubernetes.io/projected/20b78882-caed-4e29-939b-af1aea73c25e-kube-api-access-ndrcw\") pod \"20b78882-caed-4e29-939b-af1aea73c25e\" (UID: \"20b78882-caed-4e29-939b-af1aea73c25e\") "
Oct 29 21:55:37.945771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.945750    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-xks-ca\") pod \"20b78882-caed-4e29-939b-af1aea73c25e\" (UID: \"20b78882-caed-4e29-939b-af1aea73c25e\") "
Oct 29 21:55:37.945848 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.945830    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-openshift-ca\") pod \"20b78882-caed-4e29-939b-af1aea73c25e\" (UID: \"20b78882-caed-4e29-939b-af1aea73c25e\") "
Oct 29 21:55:37.946216 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.946193    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-openshift-ca" pod "20b78882-caed-4e29-939b-af1aea73c25e" (UID: "20b78882-caed-4e29-939b-af1aea73c25e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:37.947084 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.947062    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:37.948470 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.948451    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-xks-ca" pod "20b78882-caed-4e29-939b-af1aea73c25e" (UID: "20b78882-caed-4e29-939b-af1aea73c25e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:37.948559 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:37.948496    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b78882-caed-4e29-939b-af1aea73c25e-kube-api-access-ndrcw" pod "20b78882-caed-4e29-939b-af1aea73c25e" (UID: "20b78882-caed-4e29-939b-af1aea73c25e"). InnerVolumeSpecName "kube-api-access-ndrcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:38.032549 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.032446    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz" podUID="89951ee0-b7e8-48dd-8441-4da86c0fbc62" containerName="payload-pre-processing" containerID="cri-o://3239ecce0640c66f8d933b2b25285766b41bb5708b2d44d72295570b0d689fe7" gracePeriod=25
Oct 29 21:55:38.048103 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.048072    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndrcw\" (UniqueName: \"kubernetes.io/projected/20b78882-caed-4e29-939b-af1aea73c25e-kube-api-access-ndrcw\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:38.048209 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.048110    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20b78882-caed-4e29-939b-af1aea73c25e-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:38.431534 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.431507    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"
Oct 29 21:55:38.461028 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.461000    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89951ee0-b7e8-48dd-8441-4da86c0fbc62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89951ee0-b7e8-48dd-8441-4da86c0fbc62-plugins-config-volume\") pod \"89951ee0-b7e8-48dd-8441-4da86c0fbc62\" (UID: \"89951ee0-b7e8-48dd-8441-4da86c0fbc62\") "
Oct 29 21:55:38.461207 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.461093    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89951ee0-b7e8-48dd-8441-4da86c0fbc62-kube-api-access-7tx2k\" (UniqueName: \"kubernetes.io/projected/89951ee0-b7e8-48dd-8441-4da86c0fbc62-kube-api-access-7tx2k\") pod \"89951ee0-b7e8-48dd-8441-4da86c0fbc62\" (UID: \"89951ee0-b7e8-48dd-8441-4da86c0fbc62\") "
Oct 29 21:55:38.461444 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.461418    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89951ee0-b7e8-48dd-8441-4da86c0fbc62-plugins-config-volume" pod "89951ee0-b7e8-48dd-8441-4da86c0fbc62" (UID: "89951ee0-b7e8-48dd-8441-4da86c0fbc62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:38.461544 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.461529    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89951ee0-b7e8-48dd-8441-4da86c0fbc62-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:38.463550 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.463520    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89951ee0-b7e8-48dd-8441-4da86c0fbc62-kube-api-access-7tx2k" pod "89951ee0-b7e8-48dd-8441-4da86c0fbc62" (UID: "89951ee0-b7e8-48dd-8441-4da86c0fbc62"). InnerVolumeSpecName "kube-api-access-7tx2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:38.539489 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.539442    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr" event={"ID":"20b78882-caed-4e29-939b-af1aea73c25e","Type":"ContainerDied","Data":"df8912961960a0ce7dc9a8669b926b211f9e6ad4b2721d2caab024152fd235b3"}
Oct 29 21:55:38.539489 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.539492    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"
Oct 29 21:55:38.539784 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.539549    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df8912961960a0ce7dc9a8669b926b211f9e6ad4b2721d2caab024152fd235b3"
Oct 29 21:55:38.546980 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.546914    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" event={"ID":"84119759-f427-4a8d-960e-8b342fca0d38","Type":"ContainerStarted","Data":"8824abb321ebecc9d637b050ea183c5a3bc42a559dc32b2e6e0254c2320b6290"}
Oct 29 21:55:38.547909 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.547746    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" event={"ID":"84119759-f427-4a8d-960e-8b342fca0d38","Type":"ContainerStarted","Data":"6d8aea89d5dc134774fe3387d432b32e4145bacea1f2a038553655688099d161"}
Oct 29 21:55:38.548591 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.548551    7082 generic.go:356] "Generic (PLEG): container finished" podID="89951ee0-b7e8-48dd-8441-4da86c0fbc62" containerID="3239ecce0640c66f8d933b2b25285766b41bb5708b2d44d72295570b0d689fe7" exitCode=0
Oct 29 21:55:38.548735 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.548718    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"
Oct 29 21:55:38.548787 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.548730    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz" event={"ID":"89951ee0-b7e8-48dd-8441-4da86c0fbc62","Type":"ContainerDied","Data":"3239ecce0640c66f8d933b2b25285766b41bb5708b2d44d72295570b0d689fe7"}
Oct 29 21:55:38.548787 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.548773    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz" event={"ID":"89951ee0-b7e8-48dd-8441-4da86c0fbc62","Type":"ContainerDied","Data":"9d9e979301927f363b175873b91e5c6c58457bcab50dfd22ca94051fe46052be"}
Oct 29 21:55:38.548871 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.548803    7082 scope.go:122] "RemoveContainer" containerID="3239ecce0640c66f8d933b2b25285766b41bb5708b2d44d72295570b0d689fe7"
Oct 29 21:55:38.563380 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.563338    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tx2k\" (UniqueName: \"kubernetes.io/projected/89951ee0-b7e8-48dd-8441-4da86c0fbc62-kube-api-access-7tx2k\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:38.566945 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.566912    7082 scope.go:122] "RemoveContainer" containerID="3239ecce0640c66f8d933b2b25285766b41bb5708b2d44d72295570b0d689fe7"
Oct 29 21:55:38.567267 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:55:38.567248    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3239ecce0640c66f8d933b2b25285766b41bb5708b2d44d72295570b0d689fe7\": container with ID starting with 3239ecce0640c66f8d933b2b25285766b41bb5708b2d44d72295570b0d689fe7 not found: ID does not exist" containerID="3239ecce0640c66f8d933b2b25285766b41bb5708b2d44d72295570b0d689fe7"
Oct 29 21:55:38.567321 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.567277    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3239ecce0640c66f8d933b2b25285766b41bb5708b2d44d72295570b0d689fe7"} err="failed to get container status \"3239ecce0640c66f8d933b2b25285766b41bb5708b2d44d72295570b0d689fe7\": rpc error: code = NotFound desc = could not find container \"3239ecce0640c66f8d933b2b25285766b41bb5708b2d44d72295570b0d689fe7\": container with ID starting with 3239ecce0640c66f8d933b2b25285766b41bb5708b2d44d72295570b0d689fe7 not found: ID does not exist"
Oct 29 21:55:38.612397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.611026    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podStartSLOduration=1.611006266 podStartE2EDuration="1.611006266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:38.596113912 +0000 UTC m=+511.096156212" watchObservedRunningTime="2026-10-29 21:55:38.611006266 +0000 UTC m=+511.111048567"
Oct 29 21:55:38.653986 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.653896    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"]
Oct 29 21:55:38.656244 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:38.656225    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-64505fc5-69fd8b758c-xkvbz"]
Oct 29 21:55:39.428973 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:39.428942    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:39.442022 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:39.441994    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:39.558590 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:39.558560    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:39.559552 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:39.559531    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:55:39.618087 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:39.618034    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"]
Oct 29 21:55:39.634679 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:39.634252    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89951ee0-b7e8-48dd-8441-4da86c0fbc62" path="/var/lib/kubelet/pods/89951ee0-b7e8-48dd-8441-4da86c0fbc62/volumes"
Oct 29 21:55:39.639581 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:39.639555    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"]
Oct 29 21:55:41.476437 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:41.476347    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"]
Oct 29 21:55:41.506182 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:41.506118    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f5bae148-bx7p6"]
Oct 29 21:55:41.572727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:41.572687    7082 generic.go:356] "Generic (PLEG): container finished" podID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerID="fc03b2c343d00572920143694606e317b96698aae6680b9a769738f78870d558" exitCode=0
Oct 29 21:55:41.572894 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:41.572800    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" event={"ID":"ad3feeed-2094-44d9-aad9-00d17e6bbef0","Type":"ContainerDied","Data":"fc03b2c343d00572920143694606e317b96698aae6680b9a769738f78870d558"}
Oct 29 21:55:41.573267 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:41.573246    7082 scope.go:122] "RemoveContainer" containerID="fc03b2c343d00572920143694606e317b96698aae6680b9a769738f78870d558"
Oct 29 21:55:41.574441 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:41.574415    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:41.574500 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:41.574486    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:41.574541 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:41.574518    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:41.635121 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:41.635074    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6" path="/var/lib/kubelet/pods/fd3ea07b-15c1-4ba9-b1ba-bfa8f3c279a6/volumes"
Oct 29 21:55:42.580757 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:42.580728    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" event={"ID":"ad3feeed-2094-44d9-aad9-00d17e6bbef0","Type":"ContainerStarted","Data":"0f90bf532e3c9e6d58a97f33019097792de223d14de04a57ebf4b4d9e2dfcc7e"}
Oct 29 21:55:43.347067 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:43.347035    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:43.347392 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:43.347366    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 21:55:43.347449 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:43.347426    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 21:55:44.023653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:44.023320    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"]
Oct 29 21:55:44.056008 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:44.055974    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"]
Oct 29 21:55:44.347023 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:44.346993    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 21:55:44.347185 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:44.347053    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 21:55:44.621859 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:44.621745    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5" podUID="97afdabd-be6a-4a98-ad49-77909430ed98" containerName="payload-processing" containerID="cri-o://058b442e8a4f4948f8cbd2e9dfa8cf6b5f2594fc2419fc71305cd8b7c35ea910" gracePeriod=25
Oct 29 21:55:44.640443 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:44.640057    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct" podUID="efd56092-21a8-45fa-9cac-9a6094fc5a50" containerName="payload-pre-processing" containerID="cri-o://7485a4dfd6d05ada65551d7139ef89aeb8e6c108c5fa508b0d0d070548e5b5fa" gracePeriod=25
Oct 29 21:55:44.642774 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:44.642752    7082 logging.go:55] [core] [Channel #68 SubChannel #69]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.56:50052", ServerName: "10.134.0.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.56:50052: connect: connection refused"
Oct 29 21:55:45.051860 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.050953    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"
Oct 29 21:55:45.128405 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.128261    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd56092-21a8-45fa-9cac-9a6094fc5a50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd56092-21a8-45fa-9cac-9a6094fc5a50-plugins-config-volume\") pod \"efd56092-21a8-45fa-9cac-9a6094fc5a50\" (UID: \"efd56092-21a8-45fa-9cac-9a6094fc5a50\") "
Oct 29 21:55:45.128405 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.128313    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd56092-21a8-45fa-9cac-9a6094fc5a50-kube-api-access-zct4w\" (UniqueName: \"kubernetes.io/projected/efd56092-21a8-45fa-9cac-9a6094fc5a50-kube-api-access-zct4w\") pod \"efd56092-21a8-45fa-9cac-9a6094fc5a50\" (UID: \"efd56092-21a8-45fa-9cac-9a6094fc5a50\") "
Oct 29 21:55:45.128760 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.128732    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd56092-21a8-45fa-9cac-9a6094fc5a50-plugins-config-volume" pod "efd56092-21a8-45fa-9cac-9a6094fc5a50" (UID: "efd56092-21a8-45fa-9cac-9a6094fc5a50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:45.128987 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.128962    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"
Oct 29 21:55:45.129272 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.128966    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efd56092-21a8-45fa-9cac-9a6094fc5a50-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:45.130975 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.130945    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd56092-21a8-45fa-9cac-9a6094fc5a50-kube-api-access-zct4w" pod "efd56092-21a8-45fa-9cac-9a6094fc5a50" (UID: "efd56092-21a8-45fa-9cac-9a6094fc5a50"). InnerVolumeSpecName "kube-api-access-zct4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:45.232725 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.230102    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97afdabd-be6a-4a98-ad49-77909430ed98-kube-api-access-ctmlm\" (UniqueName: \"kubernetes.io/projected/97afdabd-be6a-4a98-ad49-77909430ed98-kube-api-access-ctmlm\") pod \"97afdabd-be6a-4a98-ad49-77909430ed98\" (UID: \"97afdabd-be6a-4a98-ad49-77909430ed98\") "
Oct 29 21:55:45.232725 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.230164    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97afdabd-be6a-4a98-ad49-77909430ed98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97afdabd-be6a-4a98-ad49-77909430ed98-plugins-config-volume\") pod \"97afdabd-be6a-4a98-ad49-77909430ed98\" (UID: \"97afdabd-be6a-4a98-ad49-77909430ed98\") "
Oct 29 21:55:45.232725 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.230478    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zct4w\" (UniqueName: \"kubernetes.io/projected/efd56092-21a8-45fa-9cac-9a6094fc5a50-kube-api-access-zct4w\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:45.232725 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.230849    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97afdabd-be6a-4a98-ad49-77909430ed98-plugins-config-volume" pod "97afdabd-be6a-4a98-ad49-77909430ed98" (UID: "97afdabd-be6a-4a98-ad49-77909430ed98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:45.233194 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.233171    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97afdabd-be6a-4a98-ad49-77909430ed98-kube-api-access-ctmlm" pod "97afdabd-be6a-4a98-ad49-77909430ed98" (UID: "97afdabd-be6a-4a98-ad49-77909430ed98"). InnerVolumeSpecName "kube-api-access-ctmlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:45.331480 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.331441    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctmlm\" (UniqueName: \"kubernetes.io/projected/97afdabd-be6a-4a98-ad49-77909430ed98-kube-api-access-ctmlm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:45.331480 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.331474    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97afdabd-be6a-4a98-ad49-77909430ed98-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:45.347343 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.347316    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 21:55:45.347481 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.347373    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 21:55:45.643916 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.643869    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5" podUID="97afdabd-be6a-4a98-ad49-77909430ed98" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.56:50052\" within 1s: context deadline exceeded"
Oct 29 21:55:45.647493 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.647459    7082 generic.go:356] "Generic (PLEG): container finished" podID="efd56092-21a8-45fa-9cac-9a6094fc5a50" containerID="7485a4dfd6d05ada65551d7139ef89aeb8e6c108c5fa508b0d0d070548e5b5fa" exitCode=0
Oct 29 21:55:45.647603 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.647508    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"
Oct 29 21:55:45.647780 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.647755    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct" event={"ID":"efd56092-21a8-45fa-9cac-9a6094fc5a50","Type":"ContainerDied","Data":"7485a4dfd6d05ada65551d7139ef89aeb8e6c108c5fa508b0d0d070548e5b5fa"}
Oct 29 21:55:45.647845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.647809    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct" event={"ID":"efd56092-21a8-45fa-9cac-9a6094fc5a50","Type":"ContainerDied","Data":"7bc2da2f08dc1f46e3b2ed2c9074c43153509e6ab741c7bf9550ae5ad3f32f1e"}
Oct 29 21:55:45.647894 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.647842    7082 scope.go:122] "RemoveContainer" containerID="7485a4dfd6d05ada65551d7139ef89aeb8e6c108c5fa508b0d0d070548e5b5fa"
Oct 29 21:55:45.650227 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.650184    7082 generic.go:356] "Generic (PLEG): container finished" podID="97afdabd-be6a-4a98-ad49-77909430ed98" containerID="058b442e8a4f4948f8cbd2e9dfa8cf6b5f2594fc2419fc71305cd8b7c35ea910" exitCode=0
Oct 29 21:55:45.650304 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.650242    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5" event={"ID":"97afdabd-be6a-4a98-ad49-77909430ed98","Type":"ContainerDied","Data":"058b442e8a4f4948f8cbd2e9dfa8cf6b5f2594fc2419fc71305cd8b7c35ea910"}
Oct 29 21:55:45.650304 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.650283    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5" event={"ID":"97afdabd-be6a-4a98-ad49-77909430ed98","Type":"ContainerDied","Data":"01f3a58ca71abcf1f817b82ce04acc51355b2e281eb9caefae1a70abc2ba02a6"}
Oct 29 21:55:45.650466 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.650449    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"
Oct 29 21:55:45.665690 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.665661    7082 scope.go:122] "RemoveContainer" containerID="7485a4dfd6d05ada65551d7139ef89aeb8e6c108c5fa508b0d0d070548e5b5fa"
Oct 29 21:55:45.666002 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:55:45.665985    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7485a4dfd6d05ada65551d7139ef89aeb8e6c108c5fa508b0d0d070548e5b5fa\": container with ID starting with 7485a4dfd6d05ada65551d7139ef89aeb8e6c108c5fa508b0d0d070548e5b5fa not found: ID does not exist" containerID="7485a4dfd6d05ada65551d7139ef89aeb8e6c108c5fa508b0d0d070548e5b5fa"
Oct 29 21:55:45.666093 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.666077    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7485a4dfd6d05ada65551d7139ef89aeb8e6c108c5fa508b0d0d070548e5b5fa"} err="failed to get container status \"7485a4dfd6d05ada65551d7139ef89aeb8e6c108c5fa508b0d0d070548e5b5fa\": rpc error: code = NotFound desc = could not find container \"7485a4dfd6d05ada65551d7139ef89aeb8e6c108c5fa508b0d0d070548e5b5fa\": container with ID starting with 7485a4dfd6d05ada65551d7139ef89aeb8e6c108c5fa508b0d0d070548e5b5fa not found: ID does not exist"
Oct 29 21:55:45.666155 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.666147    7082 scope.go:122] "RemoveContainer" containerID="058b442e8a4f4948f8cbd2e9dfa8cf6b5f2594fc2419fc71305cd8b7c35ea910"
Oct 29 21:55:45.675178 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.675160    7082 scope.go:122] "RemoveContainer" containerID="058b442e8a4f4948f8cbd2e9dfa8cf6b5f2594fc2419fc71305cd8b7c35ea910"
Oct 29 21:55:45.675479 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:55:45.675451    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058b442e8a4f4948f8cbd2e9dfa8cf6b5f2594fc2419fc71305cd8b7c35ea910\": container with ID starting with 058b442e8a4f4948f8cbd2e9dfa8cf6b5f2594fc2419fc71305cd8b7c35ea910 not found: ID does not exist" containerID="058b442e8a4f4948f8cbd2e9dfa8cf6b5f2594fc2419fc71305cd8b7c35ea910"
Oct 29 21:55:45.675557 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.675485    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058b442e8a4f4948f8cbd2e9dfa8cf6b5f2594fc2419fc71305cd8b7c35ea910"} err="failed to get container status \"058b442e8a4f4948f8cbd2e9dfa8cf6b5f2594fc2419fc71305cd8b7c35ea910\": rpc error: code = NotFound desc = could not find container \"058b442e8a4f4948f8cbd2e9dfa8cf6b5f2594fc2419fc71305cd8b7c35ea910\": container with ID starting with 058b442e8a4f4948f8cbd2e9dfa8cf6b5f2594fc2419fc71305cd8b7c35ea910 not found: ID does not exist"
Oct 29 21:55:45.718014 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.717962    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"]
Oct 29 21:55:45.719614 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.719586    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-de7186ef-8ff4b74c9-lg9ct"]
Oct 29 21:55:45.765989 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.765955    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"]
Oct 29 21:55:45.770324 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:45.770301    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-de7186ef-69d5748958-9fnw5"]
Oct 29 21:55:46.346913 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:46.346878    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 21:55:46.347423 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:46.346934    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 21:55:47.022065 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:47.022030    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"]
Oct 29 21:55:47.033898 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:47.033864    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de7186ef-g479m"]
Oct 29 21:55:47.241236 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:47.240916    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"]
Oct 29 21:55:47.241236 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:47.241176    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy" containerID="cri-o://e2352975cddf63b67507a6d3411d0c1e3884dfadb19450286cf2696e664b9c65" gracePeriod=30
Oct 29 21:55:47.359652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:47.350170    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 29 21:55:47.359652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:47.350297    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 29 21:55:47.537405 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:47.536544    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"]
Oct 29 21:55:47.539495 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:47.539461    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f97668b9-mdq7x"]
Oct 29 21:55:47.636919 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:47.636846    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895a41f7-ce04-428c-9ba8-0afc0823aa88" path="/var/lib/kubelet/pods/895a41f7-ce04-428c-9ba8-0afc0823aa88/volumes"
Oct 29 21:55:47.639037 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:47.637849    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97afdabd-be6a-4a98-ad49-77909430ed98" path="/var/lib/kubelet/pods/97afdabd-be6a-4a98-ad49-77909430ed98/volumes"
Oct 29 21:55:47.639037 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:47.638319    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af7a0574-17ca-4088-9470-d151a6177705" path="/var/lib/kubelet/pods/af7a0574-17ca-4088-9470-d151a6177705/volumes"
Oct 29 21:55:47.639037 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:47.638693    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd56092-21a8-45fa-9cac-9a6094fc5a50" path="/var/lib/kubelet/pods/efd56092-21a8-45fa-9cac-9a6094fc5a50/volumes"
Oct 29 21:55:48.089139 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.089101    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"]
Oct 29 21:55:48.089978 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.089951    7082 cpu_manager.go:414] "containerMap: removing container" podUID="97afdabd-be6a-4a98-ad49-77909430ed98" containerName="payload-processing"
Oct 29 21:55:48.089978 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.089980    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97afdabd-be6a-4a98-ad49-77909430ed98" containerName="payload-processing"
Oct 29 21:55:48.090130 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.090021    7082 cpu_manager.go:414] "containerMap: removing container" podUID="efd56092-21a8-45fa-9cac-9a6094fc5a50" containerName="payload-pre-processing"
Oct 29 21:55:48.090130 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.090030    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd56092-21a8-45fa-9cac-9a6094fc5a50" containerName="payload-pre-processing"
Oct 29 21:55:48.090130 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.090056    7082 cpu_manager.go:414] "containerMap: removing container" podUID="20b78882-caed-4e29-939b-af1aea73c25e" containerName="revoke-keys"
Oct 29 21:55:48.090130 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.090064    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b78882-caed-4e29-939b-af1aea73c25e" containerName="revoke-keys"
Oct 29 21:55:48.090130 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.090084    7082 cpu_manager.go:414] "containerMap: removing container" podUID="89951ee0-b7e8-48dd-8441-4da86c0fbc62" containerName="payload-pre-processing"
Oct 29 21:55:48.090130 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.090092    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89951ee0-b7e8-48dd-8441-4da86c0fbc62" containerName="payload-pre-processing"
Oct 29 21:55:48.090316 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.090216    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="97afdabd-be6a-4a98-ad49-77909430ed98" containerName="payload-processing"
Oct 29 21:55:48.090316 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.090235    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd56092-21a8-45fa-9cac-9a6094fc5a50" containerName="payload-pre-processing"
Oct 29 21:55:48.090316 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.090249    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b78882-caed-4e29-939b-af1aea73c25e" containerName="revoke-keys"
Oct 29 21:55:48.090316 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.090259    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="89951ee0-b7e8-48dd-8441-4da86c0fbc62" containerName="payload-pre-processing"
Oct 29 21:55:48.097123 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.097091    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.114397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.112449    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a3c02e\"/\"openshift-service-ca.crt\""
Oct 29 21:55:48.115197 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.115166    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"]
Oct 29 21:55:48.124700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.123219    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a3c02e\"/\"simulator-w1-a3c02e-kserve-self-signed-certs\""
Oct 29 21:55:48.124700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.123237    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a3c02e\"/\"kube-root-ca.crt\""
Oct 29 21:55:48.124700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.123283    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a3c02e\"/\"default-dockercfg-5krsk\""
Oct 29 21:55:48.176407 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.176378    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"]
Oct 29 21:55:48.176617 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.176595    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerName="istio-proxy" containerID="cri-o://6ef82de030284440b0dc4fd91d63c1b0aff041ca6e12711453e4fa3106957132" gracePeriod=30
Oct 29 21:55:48.268306 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.268240    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7s4j\" (UniqueName: \"kubernetes.io/projected/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-kube-api-access-h7s4j\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.268306 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.268294    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tls-certs\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.268539 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.268426    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-model-cache\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.268652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.268596    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-dshm\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.268730 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.268648    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tmp-dir\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.268800 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.268775    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-home\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.351526 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.351443    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:48.351890 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.351851    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:48.352857 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.352834    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 21:55:48.369768 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.369738    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7s4j\" (UniqueName: \"kubernetes.io/projected/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-kube-api-access-h7s4j\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.370206 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.369798    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tls-certs\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.370206 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.369855    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-model-cache\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.370206 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.369997    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-dshm\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.370206 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.370024    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tmp-dir\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.370206 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.370065    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-home\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.370645 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.370600    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-home\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.370732 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.370668    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tmp-dir\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.370795 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.370767    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-model-cache\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.372677 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.372657    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-dshm\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.373258 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.373241    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tls-certs\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.397035 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.396993    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7s4j\" (UniqueName: \"kubernetes.io/projected/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-kube-api-access-h7s4j\") pod \"simulator-w1-a3c02e-kserve-84566984db-4bfmh\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") " pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.419429 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.419395    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:48.677838 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.677759    7082 generic.go:356] "Generic (PLEG): container finished" podID="ddab3d72-73bf-4033-be01-6177dddff66a" containerID="49b3ee64fd31e713c9cdb45735adec0534d19db33d957d3c3794824a8536c8a5" exitCode=0
Oct 29 21:55:48.677975 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.677834    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" event={"ID":"ddab3d72-73bf-4033-be01-6177dddff66a","Type":"ContainerDied","Data":"49b3ee64fd31e713c9cdb45735adec0534d19db33d957d3c3794824a8536c8a5"}
Oct 29 21:55:48.678297 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.678273    7082 scope.go:122] "RemoveContainer" containerID="49b3ee64fd31e713c9cdb45735adec0534d19db33d957d3c3794824a8536c8a5"
Oct 29 21:55:48.679525 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.679498    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:48.679591 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.679563    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:48.679638 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.679598    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:48.804303 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:48.804266    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"]
Oct 29 21:55:48.810668 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:48.810602    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2ccb6e6_e9bf_4bb0_9a4e_c27b1ae52f5b.slice/crio-5c809cdf0ada58600ad854985aa7b6f23ddc3c19de66f7989b770d122d40b832 WatchSource:0}: Error finding container 5c809cdf0ada58600ad854985aa7b6f23ddc3c19de66f7989b770d122d40b832: Status 404 returned error can't find the container with id 5c809cdf0ada58600ad854985aa7b6f23ddc3c19de66f7989b770d122d40b832
Oct 29 21:55:49.024840 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.024746    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g" podUID="9e80dce6-46ef-46c9-84cd-677cd9e618f0" containerName="payload-processing" containerID="cri-o://0586736d83b0044ed423a5169bc57e99081396046408e0cc2cafa027c680d127" gracePeriod=25
Oct 29 21:55:49.056706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.056675    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx" podUID="12f60c90-fa0c-4d19-9e5f-42c768a911f4" containerName="payload-pre-processing" containerID="cri-o://9cbe1943bd64ec1dc7b3a9d225452fc316b608d75f190488c16d5c5170ceaaae" gracePeriod=25
Oct 29 21:55:49.357058 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.357033    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"
Oct 29 21:55:49.370201 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.370178    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"
Oct 29 21:55:49.491652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.485400    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12f60c90-fa0c-4d19-9e5f-42c768a911f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f60c90-fa0c-4d19-9e5f-42c768a911f4-plugins-config-volume\") pod \"12f60c90-fa0c-4d19-9e5f-42c768a911f4\" (UID: \"12f60c90-fa0c-4d19-9e5f-42c768a911f4\") "
Oct 29 21:55:49.491652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.485467    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f60c90-fa0c-4d19-9e5f-42c768a911f4-kube-api-access-kbzvq\" (UniqueName: \"kubernetes.io/projected/12f60c90-fa0c-4d19-9e5f-42c768a911f4-kube-api-access-kbzvq\") pod \"12f60c90-fa0c-4d19-9e5f-42c768a911f4\" (UID: \"12f60c90-fa0c-4d19-9e5f-42c768a911f4\") "
Oct 29 21:55:49.491652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.485619    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e80dce6-46ef-46c9-84cd-677cd9e618f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e80dce6-46ef-46c9-84cd-677cd9e618f0-plugins-config-volume\") pod \"9e80dce6-46ef-46c9-84cd-677cd9e618f0\" (UID: \"9e80dce6-46ef-46c9-84cd-677cd9e618f0\") "
Oct 29 21:55:49.491652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.485698    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e80dce6-46ef-46c9-84cd-677cd9e618f0-kube-api-access-lcxsf\" (UniqueName: \"kubernetes.io/projected/9e80dce6-46ef-46c9-84cd-677cd9e618f0-kube-api-access-lcxsf\") pod \"9e80dce6-46ef-46c9-84cd-677cd9e618f0\" (UID: \"9e80dce6-46ef-46c9-84cd-677cd9e618f0\") "
Oct 29 21:55:49.491652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.487043    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f60c90-fa0c-4d19-9e5f-42c768a911f4-plugins-config-volume" pod "12f60c90-fa0c-4d19-9e5f-42c768a911f4" (UID: "12f60c90-fa0c-4d19-9e5f-42c768a911f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:49.491652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.487458    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e80dce6-46ef-46c9-84cd-677cd9e618f0-plugins-config-volume" pod "9e80dce6-46ef-46c9-84cd-677cd9e618f0" (UID: "9e80dce6-46ef-46c9-84cd-677cd9e618f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:49.491652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.491476    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f60c90-fa0c-4d19-9e5f-42c768a911f4-kube-api-access-kbzvq" pod "12f60c90-fa0c-4d19-9e5f-42c768a911f4" (UID: "12f60c90-fa0c-4d19-9e5f-42c768a911f4"). InnerVolumeSpecName "kube-api-access-kbzvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:49.492169 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.491928    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e80dce6-46ef-46c9-84cd-677cd9e618f0-kube-api-access-lcxsf" pod "9e80dce6-46ef-46c9-84cd-677cd9e618f0" (UID: "9e80dce6-46ef-46c9-84cd-677cd9e618f0"). InnerVolumeSpecName "kube-api-access-lcxsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:49.613654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.593468    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f60c90-fa0c-4d19-9e5f-42c768a911f4-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:49.613654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.593505    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbzvq\" (UniqueName: \"kubernetes.io/projected/12f60c90-fa0c-4d19-9e5f-42c768a911f4-kube-api-access-kbzvq\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:49.613654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.593520    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e80dce6-46ef-46c9-84cd-677cd9e618f0-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:49.613654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.593536    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcxsf\" (UniqueName: \"kubernetes.io/projected/9e80dce6-46ef-46c9-84cd-677cd9e618f0-kube-api-access-lcxsf\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:49.647170 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.647137    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:55:49.745458 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.745419    7082 generic.go:356] "Generic (PLEG): container finished" podID="9e80dce6-46ef-46c9-84cd-677cd9e618f0" containerID="0586736d83b0044ed423a5169bc57e99081396046408e0cc2cafa027c680d127" exitCode=0
Oct 29 21:55:49.746340 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.745734    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"
Oct 29 21:55:49.746340 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.745662    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g" event={"ID":"9e80dce6-46ef-46c9-84cd-677cd9e618f0","Type":"ContainerDied","Data":"0586736d83b0044ed423a5169bc57e99081396046408e0cc2cafa027c680d127"}
Oct 29 21:55:49.746340 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.745831    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g" event={"ID":"9e80dce6-46ef-46c9-84cd-677cd9e618f0","Type":"ContainerDied","Data":"1c5111376979c599d0edf8a6e90921135f7b202ee10efc4a8772fe894e7d6d2c"}
Oct 29 21:55:49.746340 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.745863    7082 scope.go:122] "RemoveContainer" containerID="0586736d83b0044ed423a5169bc57e99081396046408e0cc2cafa027c680d127"
Oct 29 21:55:49.751823 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.750875    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" event={"ID":"ddab3d72-73bf-4033-be01-6177dddff66a","Type":"ContainerStarted","Data":"2b1a6483ffc3e7338970e087437f16f0ba436d7946c8b5f1eda54d13a8f21884"}
Oct 29 21:55:49.760877 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.754320    7082 generic.go:356] "Generic (PLEG): container finished" podID="12f60c90-fa0c-4d19-9e5f-42c768a911f4" containerID="9cbe1943bd64ec1dc7b3a9d225452fc316b608d75f190488c16d5c5170ceaaae" exitCode=0
Oct 29 21:55:49.762125 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.762037    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"
Oct 29 21:55:49.768648 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.763518    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx" event={"ID":"12f60c90-fa0c-4d19-9e5f-42c768a911f4","Type":"ContainerDied","Data":"9cbe1943bd64ec1dc7b3a9d225452fc316b608d75f190488c16d5c5170ceaaae"}
Oct 29 21:55:49.768648 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.763582    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx" event={"ID":"12f60c90-fa0c-4d19-9e5f-42c768a911f4","Type":"ContainerDied","Data":"7ad5b1c8082f1c91aac1dd3c87eac770bd4f6fd9c166d8384e1596b2a40b8bf7"}
Oct 29 21:55:49.778456 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.768845    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" event={"ID":"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b","Type":"ContainerStarted","Data":"670529ce9d5bcd94f25e790d10960dc5cbd5ebc47e797d59329c8ea6c9bff676"}
Oct 29 21:55:49.778456 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.768925    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" event={"ID":"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b","Type":"ContainerStarted","Data":"5c809cdf0ada58600ad854985aa7b6f23ddc3c19de66f7989b770d122d40b832"}
Oct 29 21:55:49.827059 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.825395    7082 scope.go:122] "RemoveContainer" containerID="0586736d83b0044ed423a5169bc57e99081396046408e0cc2cafa027c680d127"
Oct 29 21:55:49.827059 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:55:49.825971    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0586736d83b0044ed423a5169bc57e99081396046408e0cc2cafa027c680d127\": container with ID starting with 0586736d83b0044ed423a5169bc57e99081396046408e0cc2cafa027c680d127 not found: ID does not exist" containerID="0586736d83b0044ed423a5169bc57e99081396046408e0cc2cafa027c680d127"
Oct 29 21:55:49.827059 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.826022    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0586736d83b0044ed423a5169bc57e99081396046408e0cc2cafa027c680d127"} err="failed to get container status \"0586736d83b0044ed423a5169bc57e99081396046408e0cc2cafa027c680d127\": rpc error: code = NotFound desc = could not find container \"0586736d83b0044ed423a5169bc57e99081396046408e0cc2cafa027c680d127\": container with ID starting with 0586736d83b0044ed423a5169bc57e99081396046408e0cc2cafa027c680d127 not found: ID does not exist"
Oct 29 21:55:49.827059 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.826049    7082 scope.go:122] "RemoveContainer" containerID="9cbe1943bd64ec1dc7b3a9d225452fc316b608d75f190488c16d5c5170ceaaae"
Oct 29 21:55:49.944204 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.944177    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"]
Oct 29 21:55:49.946127 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.946018    7082 scope.go:122] "RemoveContainer" containerID="9cbe1943bd64ec1dc7b3a9d225452fc316b608d75f190488c16d5c5170ceaaae"
Oct 29 21:55:49.946698 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:55:49.946604    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cbe1943bd64ec1dc7b3a9d225452fc316b608d75f190488c16d5c5170ceaaae\": container with ID starting with 9cbe1943bd64ec1dc7b3a9d225452fc316b608d75f190488c16d5c5170ceaaae not found: ID does not exist" containerID="9cbe1943bd64ec1dc7b3a9d225452fc316b608d75f190488c16d5c5170ceaaae"
Oct 29 21:55:49.946698 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.946658    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cbe1943bd64ec1dc7b3a9d225452fc316b608d75f190488c16d5c5170ceaaae"} err="failed to get container status \"9cbe1943bd64ec1dc7b3a9d225452fc316b608d75f190488c16d5c5170ceaaae\": rpc error: code = NotFound desc = could not find container \"9cbe1943bd64ec1dc7b3a9d225452fc316b608d75f190488c16d5c5170ceaaae\": container with ID starting with 9cbe1943bd64ec1dc7b3a9d225452fc316b608d75f190488c16d5c5170ceaaae not found: ID does not exist"
Oct 29 21:55:49.967688 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.957270    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8fba532a-d4b467f6c-q8hgx"]
Oct 29 21:55:49.980659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.977253    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podStartSLOduration=1.9772307059999998 podStartE2EDuration="1.977230706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:49.939378647 +0000 UTC m=+522.439420945" watchObservedRunningTime="2026-10-29 21:55:49.977230706 +0000 UTC m=+522.477273006"
Oct 29 21:55:49.992798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.992764    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"]
Oct 29 21:55:49.993046 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:49.993033    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8fba532a-5d6fd94964-2xn2g"]
Oct 29 21:55:50.375766 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.375662    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:50.376188 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.375836    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 29 21:55:50.376188 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.375879    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 29 21:55:50.428592 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.428533    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:50.467262 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.467228    7082 patch_prober.go:29] interesting pod/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 29 21:55:50.467460 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.467307    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 21:55:50.878606 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.878574    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"]
Oct 29 21:55:50.879078 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.879064    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9e80dce6-46ef-46c9-84cd-677cd9e618f0" containerName="payload-processing"
Oct 29 21:55:50.879078 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.879078    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e80dce6-46ef-46c9-84cd-677cd9e618f0" containerName="payload-processing"
Oct 29 21:55:50.879192 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.879099    7082 cpu_manager.go:414] "containerMap: removing container" podUID="12f60c90-fa0c-4d19-9e5f-42c768a911f4" containerName="payload-pre-processing"
Oct 29 21:55:50.879192 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.879105    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f60c90-fa0c-4d19-9e5f-42c768a911f4" containerName="payload-pre-processing"
Oct 29 21:55:50.879265 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.879196    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e80dce6-46ef-46c9-84cd-677cd9e618f0" containerName="payload-processing"
Oct 29 21:55:50.879265 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.879207    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f60c90-fa0c-4d19-9e5f-42c768a911f4" containerName="payload-pre-processing"
Oct 29 21:55:50.883672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.883653    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"
Oct 29 21:55:50.938287 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.934249    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b31f3f\""
Oct 29 21:55:50.938287 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.934425    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b31f3f-dockercfg-hbsrx\""
Oct 29 21:55:50.938287 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.936494    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmd27\" (UniqueName: \"kubernetes.io/projected/0369fe15-bc84-442e-973c-fde55e95785a-kube-api-access-qmd27\") pod \"payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx\" (UID: \"0369fe15-bc84-442e-973c-fde55e95785a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"
Oct 29 21:55:50.938287 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.937518    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"]
Oct 29 21:55:50.945984 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:50.945781    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0369fe15-bc84-442e-973c-fde55e95785a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx\" (UID: \"0369fe15-bc84-442e-973c-fde55e95785a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"
Oct 29 21:55:51.056869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.045195    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"]
Oct 29 21:55:51.064328 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.064282    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0369fe15-bc84-442e-973c-fde55e95785a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx\" (UID: \"0369fe15-bc84-442e-973c-fde55e95785a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"
Oct 29 21:55:51.065894 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.065860    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmd27\" (UniqueName: \"kubernetes.io/projected/0369fe15-bc84-442e-973c-fde55e95785a-kube-api-access-qmd27\") pod \"payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx\" (UID: \"0369fe15-bc84-442e-973c-fde55e95785a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"
Oct 29 21:55:51.066411 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.065708    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0369fe15-bc84-442e-973c-fde55e95785a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx\" (UID: \"0369fe15-bc84-442e-973c-fde55e95785a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"
Oct 29 21:55:51.068834 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.068794    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"
Oct 29 21:55:51.069586 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.069565    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"]
Oct 29 21:55:51.134691 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.134345    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmd27\" (UniqueName: \"kubernetes.io/projected/0369fe15-bc84-442e-973c-fde55e95785a-kube-api-access-qmd27\") pod \"payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx\" (UID: \"0369fe15-bc84-442e-973c-fde55e95785a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"
Oct 29 21:55:51.168658 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.168612    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7wz\" (UniqueName: \"kubernetes.io/projected/efe88fdd-8cf4-4758-8d81-546d50c2786f-kube-api-access-kq7wz\") pod \"payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2\" (UID: \"efe88fdd-8cf4-4758-8d81-546d50c2786f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"
Oct 29 21:55:51.168956 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.168934    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe88fdd-8cf4-4758-8d81-546d50c2786f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2\" (UID: \"efe88fdd-8cf4-4758-8d81-546d50c2786f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"
Oct 29 21:55:51.270797 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.270760    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7wz\" (UniqueName: \"kubernetes.io/projected/efe88fdd-8cf4-4758-8d81-546d50c2786f-kube-api-access-kq7wz\") pod \"payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2\" (UID: \"efe88fdd-8cf4-4758-8d81-546d50c2786f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"
Oct 29 21:55:51.270978 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.270844    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe88fdd-8cf4-4758-8d81-546d50c2786f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2\" (UID: \"efe88fdd-8cf4-4758-8d81-546d50c2786f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"
Oct 29 21:55:51.271605 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.271571    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe88fdd-8cf4-4758-8d81-546d50c2786f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2\" (UID: \"efe88fdd-8cf4-4758-8d81-546d50c2786f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"
Oct 29 21:55:51.280509 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.280481    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"
Oct 29 21:55:51.318325 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.318292    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7wz\" (UniqueName: \"kubernetes.io/projected/efe88fdd-8cf4-4758-8d81-546d50c2786f-kube-api-access-kq7wz\") pod \"payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2\" (UID: \"efe88fdd-8cf4-4758-8d81-546d50c2786f\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"
Oct 29 21:55:51.376007 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.375969    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 29 21:55:51.376460 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.376044    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 29 21:55:51.439151 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.438866    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"
Oct 29 21:55:51.637795 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.637755    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12f60c90-fa0c-4d19-9e5f-42c768a911f4" path="/var/lib/kubelet/pods/12f60c90-fa0c-4d19-9e5f-42c768a911f4/volumes"
Oct 29 21:55:51.638377 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.638356    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e80dce6-46ef-46c9-84cd-677cd9e618f0" path="/var/lib/kubelet/pods/9e80dce6-46ef-46c9-84cd-677cd9e618f0/volumes"
Oct 29 21:55:51.639573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.638800    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"]
Oct 29 21:55:51.646395 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:51.645825    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0369fe15_bc84_442e_973c_fde55e95785a.slice/crio-16f13b5e73624452f515e77cc4729901f3c92f5ad8ccd55081e00e497e2a326f WatchSource:0}: Error finding container 16f13b5e73624452f515e77cc4729901f3c92f5ad8ccd55081e00e497e2a326f: Status 404 returned error can't find the container with id 16f13b5e73624452f515e77cc4729901f3c92f5ad8ccd55081e00e497e2a326f
Oct 29 21:55:51.823687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.810696    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx" event={"ID":"0369fe15-bc84-442e-973c-fde55e95785a","Type":"ContainerStarted","Data":"6739c9b6994f6d5bc303ce2ed054bc240cb72d5c53445eb5677cd69e878a61f5"}
Oct 29 21:55:51.823687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.810753    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx" event={"ID":"0369fe15-bc84-442e-973c-fde55e95785a","Type":"ContainerStarted","Data":"16f13b5e73624452f515e77cc4729901f3c92f5ad8ccd55081e00e497e2a326f"}
Oct 29 21:55:51.823687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.811012    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"
Oct 29 21:55:51.823687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.823332    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"]
Oct 29 21:55:51.829062 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:51.829021    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe88fdd_8cf4_4758_8d81_546d50c2786f.slice/crio-418e3a32ff266ea48e4dc95898e6651460716017a6211f67d929fb1c75fab2ce WatchSource:0}: Error finding container 418e3a32ff266ea48e4dc95898e6651460716017a6211f67d929fb1c75fab2ce: Status 404 returned error can't find the container with id 418e3a32ff266ea48e4dc95898e6651460716017a6211f67d929fb1c75fab2ce
Oct 29 21:55:51.855113 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.853837    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"]
Oct 29 21:55:51.856916 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.856863    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx" podStartSLOduration=1.856845809 podStartE2EDuration="1.856845809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:51.854182265 +0000 UTC m=+524.354224571" watchObservedRunningTime="2026-10-29 21:55:51.856845809 +0000 UTC m=+524.356888109"
Oct 29 21:55:51.861390 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:51.861366    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8fba532a-rz9cr"]
Oct 29 21:55:52.024708 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.021541    7082 patch_prober.go:29] interesting pod/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 21:55:52.024708 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.021605    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 21:55:52.375715 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.375687    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 29 21:55:52.375868 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.375745    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 29 21:55:52.589649 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.588792    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:55:52.741062 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.740962    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-credential-socket\") pod \"05954176-b158-4666-933b-334e368ef303\" (UID: \"05954176-b158-4666-933b-334e368ef303\") "
Oct 29 21:55:52.741062 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.741037    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-istio-token\" (UniqueName: \"kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-istio-token\") pod \"05954176-b158-4666-933b-334e368ef303\" (UID: \"05954176-b158-4666-933b-334e368ef303\") "
Oct 29 21:55:52.741259 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.741074    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-data\") pod \"05954176-b158-4666-933b-334e368ef303\" (UID: \"05954176-b158-4666-933b-334e368ef303\") "
Oct 29 21:55:52.741259 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.741217    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05954176-b158-4666-933b-334e368ef303-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05954176-b158-4666-933b-334e368ef303-istiod-ca-cert\") pod \"05954176-b158-4666-933b-334e368ef303\" (UID: \"05954176-b158-4666-933b-334e368ef303\") "
Oct 29 21:55:52.741366 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.741268    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-certs\") pod \"05954176-b158-4666-933b-334e368ef303\" (UID: \"05954176-b158-4666-933b-334e368ef303\") "
Oct 29 21:55:52.741366 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.741279    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-credential-socket" pod "05954176-b158-4666-933b-334e368ef303" (UID: "05954176-b158-4666-933b-334e368ef303"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:52.741366 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.741325    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-socket\") pod \"05954176-b158-4666-933b-334e368ef303\" (UID: \"05954176-b158-4666-933b-334e368ef303\") "
Oct 29 21:55:52.741366 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.741359    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05954176-b158-4666-933b-334e368ef303-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05954176-b158-4666-933b-334e368ef303-istio-podinfo\") pod \"05954176-b158-4666-933b-334e368ef303\" (UID: \"05954176-b158-4666-933b-334e368ef303\") "
Oct 29 21:55:52.741529 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.741412    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-kube-api-access-fhmlf\" (UniqueName: \"kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-kube-api-access-fhmlf\") pod \"05954176-b158-4666-933b-334e368ef303\" (UID: \"05954176-b158-4666-933b-334e368ef303\") "
Oct 29 21:55:52.741529 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.741442    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-data" pod "05954176-b158-4666-933b-334e368ef303" (UID: "05954176-b158-4666-933b-334e368ef303"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:52.741529 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.741449    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-envoy\") pod \"05954176-b158-4666-933b-334e368ef303\" (UID: \"05954176-b158-4666-933b-334e368ef303\") "
Oct 29 21:55:52.741911 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.741885    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-socket" pod "05954176-b158-4666-933b-334e368ef303" (UID: "05954176-b158-4666-933b-334e368ef303"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:52.742034 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.741924    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:52.742034 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.741947    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:52.742110 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.742087    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05954176-b158-4666-933b-334e368ef303-istiod-ca-cert" pod "05954176-b158-4666-933b-334e368ef303" (UID: "05954176-b158-4666-933b-334e368ef303"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:52.742460 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.742431    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-certs" pod "05954176-b158-4666-933b-334e368ef303" (UID: "05954176-b158-4666-933b-334e368ef303"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:52.744538 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.744512    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-istio-token" pod "05954176-b158-4666-933b-334e368ef303" (UID: "05954176-b158-4666-933b-334e368ef303"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:52.744664 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.744566    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-kube-api-access-fhmlf" pod "05954176-b158-4666-933b-334e368ef303" (UID: "05954176-b158-4666-933b-334e368ef303"). InnerVolumeSpecName "kube-api-access-fhmlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:52.746066 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.745993    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05954176-b158-4666-933b-334e368ef303-istio-podinfo" pod "05954176-b158-4666-933b-334e368ef303" (UID: "05954176-b158-4666-933b-334e368ef303"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 21:55:52.746066 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.746032    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-envoy" pod "05954176-b158-4666-933b-334e368ef303" (UID: "05954176-b158-4666-933b-334e368ef303"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:52.825645 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.824856    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2" event={"ID":"efe88fdd-8cf4-4758-8d81-546d50c2786f","Type":"ContainerStarted","Data":"34ff6b30ffbac0f10e1d8a832bc405c73f1a8c4ac90c59517c183b0cbe080f95"}
Oct 29 21:55:52.825645 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.824919    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"
Oct 29 21:55:52.825645 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.824943    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2" event={"ID":"efe88fdd-8cf4-4758-8d81-546d50c2786f","Type":"ContainerStarted","Data":"418e3a32ff266ea48e4dc95898e6651460716017a6211f67d929fb1c75fab2ce"}
Oct 29 21:55:52.826839 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.826789    7082 generic.go:356] "Generic (PLEG): container finished" podID="05954176-b158-4666-933b-334e368ef303" containerID="e2352975cddf63b67507a6d3411d0c1e3884dfadb19450286cf2696e664b9c65" exitCode=0
Oct 29 21:55:52.826974 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.826849    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"
Oct 29 21:55:52.826974 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.826896    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" event={"ID":"05954176-b158-4666-933b-334e368ef303","Type":"ContainerDied","Data":"e2352975cddf63b67507a6d3411d0c1e3884dfadb19450286cf2696e664b9c65"}
Oct 29 21:55:52.826974 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.826941    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv" event={"ID":"05954176-b158-4666-933b-334e368ef303","Type":"ContainerDied","Data":"026b9c9ddc8b74d88cbbf7be28a589665a2a2ec1e29dd7efda38ede9572f61b1"}
Oct 29 21:55:52.826974 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.826963    7082 scope.go:122] "RemoveContainer" containerID="e2352975cddf63b67507a6d3411d0c1e3884dfadb19450286cf2696e664b9c65"
Oct 29 21:55:52.837900 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.837880    7082 scope.go:122] "RemoveContainer" containerID="c9d21e6d3dde9c1560cd50b67969b3838a9b808d9b518f976cc6382e5779bd90"
Oct 29 21:55:52.842796 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.842771    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhmlf\" (UniqueName: \"kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-kube-api-access-fhmlf\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:52.842881 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.842800    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:52.842881 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.842814    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05954176-b158-4666-933b-334e368ef303-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:52.842881 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.842826    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05954176-b158-4666-933b-334e368ef303-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:52.842881 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.842841    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:52.842881 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.842853    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05954176-b158-4666-933b-334e368ef303-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:52.842881 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.842865    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05954176-b158-4666-933b-334e368ef303-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:52.847102 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.847077    7082 scope.go:122] "RemoveContainer" containerID="e2352975cddf63b67507a6d3411d0c1e3884dfadb19450286cf2696e664b9c65"
Oct 29 21:55:52.847402 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:55:52.847383    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2352975cddf63b67507a6d3411d0c1e3884dfadb19450286cf2696e664b9c65\": container with ID starting with e2352975cddf63b67507a6d3411d0c1e3884dfadb19450286cf2696e664b9c65 not found: ID does not exist" containerID="e2352975cddf63b67507a6d3411d0c1e3884dfadb19450286cf2696e664b9c65"
Oct 29 21:55:52.847480 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.847414    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2352975cddf63b67507a6d3411d0c1e3884dfadb19450286cf2696e664b9c65"} err="failed to get container status \"e2352975cddf63b67507a6d3411d0c1e3884dfadb19450286cf2696e664b9c65\": rpc error: code = NotFound desc = could not find container \"e2352975cddf63b67507a6d3411d0c1e3884dfadb19450286cf2696e664b9c65\": container with ID starting with e2352975cddf63b67507a6d3411d0c1e3884dfadb19450286cf2696e664b9c65 not found: ID does not exist"
Oct 29 21:55:52.847480 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.847441    7082 scope.go:122] "RemoveContainer" containerID="c9d21e6d3dde9c1560cd50b67969b3838a9b808d9b518f976cc6382e5779bd90"
Oct 29 21:55:52.847742 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:55:52.847718    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d21e6d3dde9c1560cd50b67969b3838a9b808d9b518f976cc6382e5779bd90\": container with ID starting with c9d21e6d3dde9c1560cd50b67969b3838a9b808d9b518f976cc6382e5779bd90 not found: ID does not exist" containerID="c9d21e6d3dde9c1560cd50b67969b3838a9b808d9b518f976cc6382e5779bd90"
Oct 29 21:55:52.847807 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.847751    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d21e6d3dde9c1560cd50b67969b3838a9b808d9b518f976cc6382e5779bd90"} err="failed to get container status \"c9d21e6d3dde9c1560cd50b67969b3838a9b808d9b518f976cc6382e5779bd90\": rpc error: code = NotFound desc = could not find container \"c9d21e6d3dde9c1560cd50b67969b3838a9b808d9b518f976cc6382e5779bd90\": container with ID starting with c9d21e6d3dde9c1560cd50b67969b3838a9b808d9b518f976cc6382e5779bd90 not found: ID does not exist"
Oct 29 21:55:52.868613 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.868571    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2" podStartSLOduration=2.868558672 podStartE2EDuration="2.868558672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:52.867701748 +0000 UTC m=+525.367744049" watchObservedRunningTime="2026-10-29 21:55:52.868558672 +0000 UTC m=+525.368600971"
Oct 29 21:55:52.926222 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.926188    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"]
Oct 29 21:55:52.928465 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:52.928444    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-de7186ef-openshift-default-7c7d6695d9-qm8zv"]
Oct 29 21:55:53.319460 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.319429    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"]
Oct 29 21:55:53.320307 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.320279    7082 cpu_manager.go:414] "containerMap: removing container" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy"
Oct 29 21:55:53.320307 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.320304    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy"
Oct 29 21:55:53.320492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.320346    7082 cpu_manager.go:414] "containerMap: removing container" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy"
Oct 29 21:55:53.320492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.320354    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy"
Oct 29 21:55:53.320492 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.320472    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy"
Oct 29 21:55:53.323975 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.323955    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.343677 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.342414    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8a9e5f-openshift-default-dockercfg-98j45\""
Oct 29 21:55:53.346474 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.346403    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"]
Oct 29 21:55:53.348672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.348193    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.348672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.348254    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.348672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.348279    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.348672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.348304    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.348672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.348391    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.348672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.348419    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.348672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.348449    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgx49\" (UniqueName: \"kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-kube-api-access-qgx49\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.348672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.348496    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.348672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.348529    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.376245 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.376187    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 29 21:55:53.376564 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.376510    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 29 21:55:53.449771 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.449726    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.449935 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.449797    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.449935 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.449820    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.449935 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.449847    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.450115 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.449949    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.450115 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.449979    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.450115 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.450011    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgx49\" (UniqueName: \"kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-kube-api-access-qgx49\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.450115 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.450068    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.450115 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.450107    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.450549 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.450528    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.451823 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.451495    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.451823 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.451738    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.451996 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.451825    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.452072 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.452033    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.454137 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.454117    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.456658 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.455331    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.474933 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.474868    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgx49\" (UniqueName: \"kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-kube-api-access-qgx49\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.475704 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.475660    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.561335 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.561255    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:55:53.634339 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.634299    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05954176-b158-4666-933b-334e368ef303" path="/var/lib/kubelet/pods/05954176-b158-4666-933b-334e368ef303/volumes"
Oct 29 21:55:53.635072 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.635052    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b78882-caed-4e29-939b-af1aea73c25e" path="/var/lib/kubelet/pods/20b78882-caed-4e29-939b-af1aea73c25e/volumes"
Oct 29 21:55:53.640121 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.640100    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:53.652819 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.652789    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istiod-ca-cert\") pod \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") "
Oct 29 21:55:53.652941 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.652829    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-credential-socket\") pod \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") "
Oct 29 21:55:53.652941 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.652901    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-envoy\") pod \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") "
Oct 29 21:55:53.653055 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.652960    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-certs\") pod \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") "
Oct 29 21:55:53.653055 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.653024    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-podinfo\") pod \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") "
Oct 29 21:55:53.653176 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.653057    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-data\") pod \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") "
Oct 29 21:55:53.653176 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.653104    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-kube-api-access-hq4rj\" (UniqueName: \"kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-kube-api-access-hq4rj\") pod \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") "
Oct 29 21:55:53.653176 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.653141    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-socket\") pod \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") "
Oct 29 21:55:53.653443 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.653230    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-token\" (UniqueName: \"kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-token\") pod \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\" (UID: \"45bb9e0c-17cf-41f8-b79e-841fa270c02b\") "
Oct 29 21:55:53.654054 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.654030    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istiod-ca-cert" pod "45bb9e0c-17cf-41f8-b79e-841fa270c02b" (UID: "45bb9e0c-17cf-41f8-b79e-841fa270c02b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:53.654299 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.654278    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-credential-socket" pod "45bb9e0c-17cf-41f8-b79e-841fa270c02b" (UID: "45bb9e0c-17cf-41f8-b79e-841fa270c02b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:53.655277 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.654850    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-certs" pod "45bb9e0c-17cf-41f8-b79e-841fa270c02b" (UID: "45bb9e0c-17cf-41f8-b79e-841fa270c02b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:53.655277 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.655034    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-data" pod "45bb9e0c-17cf-41f8-b79e-841fa270c02b" (UID: "45bb9e0c-17cf-41f8-b79e-841fa270c02b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:53.656188 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.656167    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-socket" pod "45bb9e0c-17cf-41f8-b79e-841fa270c02b" (UID: "45bb9e0c-17cf-41f8-b79e-841fa270c02b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:53.657013 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.656975    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-podinfo" pod "45bb9e0c-17cf-41f8-b79e-841fa270c02b" (UID: "45bb9e0c-17cf-41f8-b79e-841fa270c02b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 21:55:53.657013 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.656984    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-envoy" pod "45bb9e0c-17cf-41f8-b79e-841fa270c02b" (UID: "45bb9e0c-17cf-41f8-b79e-841fa270c02b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:53.657496 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.657464    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-kube-api-access-hq4rj" pod "45bb9e0c-17cf-41f8-b79e-841fa270c02b" (UID: "45bb9e0c-17cf-41f8-b79e-841fa270c02b"). InnerVolumeSpecName "kube-api-access-hq4rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:53.657743 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.657718    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-token" pod "45bb9e0c-17cf-41f8-b79e-841fa270c02b" (UID: "45bb9e0c-17cf-41f8-b79e-841fa270c02b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:53.754567 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.754521    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:53.754567 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.754557    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:53.754567 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.754571    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq4rj\" (UniqueName: \"kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-kube-api-access-hq4rj\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:53.754819 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.754585    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:53.754819 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.754596    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:53.754819 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.754608    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:53.754819 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.754619    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:53.754819 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.754649    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:53.754819 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.754660    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb9e0c-17cf-41f8-b79e-841fa270c02b-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:55:53.835374 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.835340    7082 generic.go:356] "Generic (PLEG): container finished" podID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerID="6ef82de030284440b0dc4fd91d63c1b0aff041ca6e12711453e4fa3106957132" exitCode=0
Oct 29 21:55:53.835562 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.835390    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"
Oct 29 21:55:53.835562 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.835419    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" event={"ID":"45bb9e0c-17cf-41f8-b79e-841fa270c02b","Type":"ContainerDied","Data":"6ef82de030284440b0dc4fd91d63c1b0aff041ca6e12711453e4fa3106957132"}
Oct 29 21:55:53.835562 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.835482    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5" event={"ID":"45bb9e0c-17cf-41f8-b79e-841fa270c02b","Type":"ContainerDied","Data":"3076f9b5ba20594db6b0c32c4d39ccc6471a2a01d4b30b6aa0e93208c8de41bb"}
Oct 29 21:55:53.835562 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.835512    7082 scope.go:122] "RemoveContainer" containerID="6ef82de030284440b0dc4fd91d63c1b0aff041ca6e12711453e4fa3106957132"
Oct 29 21:55:53.846429 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.846395    7082 scope.go:122] "RemoveContainer" containerID="5d9403b6be3b05c4cff055788b4b413ba89d028bed746bd7426e56322f24983f"
Oct 29 21:55:53.855783 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.855760    7082 scope.go:122] "RemoveContainer" containerID="6ef82de030284440b0dc4fd91d63c1b0aff041ca6e12711453e4fa3106957132"
Oct 29 21:55:53.856076 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:55:53.856057    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef82de030284440b0dc4fd91d63c1b0aff041ca6e12711453e4fa3106957132\": container with ID starting with 6ef82de030284440b0dc4fd91d63c1b0aff041ca6e12711453e4fa3106957132 not found: ID does not exist" containerID="6ef82de030284440b0dc4fd91d63c1b0aff041ca6e12711453e4fa3106957132"
Oct 29 21:55:53.856129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.856084    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef82de030284440b0dc4fd91d63c1b0aff041ca6e12711453e4fa3106957132"} err="failed to get container status \"6ef82de030284440b0dc4fd91d63c1b0aff041ca6e12711453e4fa3106957132\": rpc error: code = NotFound desc = could not find container \"6ef82de030284440b0dc4fd91d63c1b0aff041ca6e12711453e4fa3106957132\": container with ID starting with 6ef82de030284440b0dc4fd91d63c1b0aff041ca6e12711453e4fa3106957132 not found: ID does not exist"
Oct 29 21:55:53.856129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.856102    7082 scope.go:122] "RemoveContainer" containerID="5d9403b6be3b05c4cff055788b4b413ba89d028bed746bd7426e56322f24983f"
Oct 29 21:55:53.856327 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:55:53.856309    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9403b6be3b05c4cff055788b4b413ba89d028bed746bd7426e56322f24983f\": container with ID starting with 5d9403b6be3b05c4cff055788b4b413ba89d028bed746bd7426e56322f24983f not found: ID does not exist" containerID="5d9403b6be3b05c4cff055788b4b413ba89d028bed746bd7426e56322f24983f"
Oct 29 21:55:53.856373 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.856334    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9403b6be3b05c4cff055788b4b413ba89d028bed746bd7426e56322f24983f"} err="failed to get container status \"5d9403b6be3b05c4cff055788b4b413ba89d028bed746bd7426e56322f24983f\": rpc error: code = NotFound desc = could not find container \"5d9403b6be3b05c4cff055788b4b413ba89d028bed746bd7426e56322f24983f\": container with ID starting with 5d9403b6be3b05c4cff055788b4b413ba89d028bed746bd7426e56322f24983f not found: ID does not exist"
Oct 29 21:55:53.956653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.955168    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"]
Oct 29 21:55:53.961267 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:53.961084    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f97668b9-openshift-default-85458b4fff-h6wb5"]
Oct 29 21:55:54.017350 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.016855    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"]
Oct 29 21:55:54.022207 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:54.022181    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6e140c_ea25_446f_a3b2_42d49e6634e5.slice/crio-f8b0ffd3297ce9fa7332d1ebc5b587d769a3b2eee31b5f5e571566ca3bc6e2ff WatchSource:0}: Error finding container f8b0ffd3297ce9fa7332d1ebc5b587d769a3b2eee31b5f5e571566ca3bc6e2ff: Status 404 returned error can't find the container with id f8b0ffd3297ce9fa7332d1ebc5b587d769a3b2eee31b5f5e571566ca3bc6e2ff
Oct 29 21:55:54.025528 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.025496    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:54.025643 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.025572    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:54.025643 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.025603    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:55:54.376188 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.375941    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 29 21:55:54.376188 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.376013    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 29 21:55:54.537075 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.537039    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"]
Oct 29 21:55:54.537564 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.537550    7082 cpu_manager.go:414] "containerMap: removing container" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerName="istio-proxy"
Oct 29 21:55:54.537607 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.537566    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerName="istio-proxy"
Oct 29 21:55:54.537607 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.537601    7082 cpu_manager.go:414] "containerMap: removing container" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerName="istio-proxy"
Oct 29 21:55:54.537607 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.537606    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerName="istio-proxy"
Oct 29 21:55:54.537735 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.537725    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerName="istio-proxy"
Oct 29 21:55:54.537788 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.537741    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="05954176-b158-4666-933b-334e368ef303" containerName="istio-proxy"
Oct 29 21:55:54.541101 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.541085    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"
Oct 29 21:55:54.561199 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.561177    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-a9def7\""
Oct 29 21:55:54.561381 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.561177    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-a9def7-dockercfg-kpx24\""
Oct 29 21:55:54.562484 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.562464    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"]
Oct 29 21:55:54.565702 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.565679    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e8c570-3e0f-40b7-9696-81b84b10e8eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x\" (UID: \"79e8c570-3e0f-40b7-9696-81b84b10e8eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"
Oct 29 21:55:54.565802 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.565792    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvldb\" (UniqueName: \"kubernetes.io/projected/79e8c570-3e0f-40b7-9696-81b84b10e8eb-kube-api-access-vvldb\") pod \"payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x\" (UID: \"79e8c570-3e0f-40b7-9696-81b84b10e8eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"
Oct 29 21:55:54.666709 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.666598    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e8c570-3e0f-40b7-9696-81b84b10e8eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x\" (UID: \"79e8c570-3e0f-40b7-9696-81b84b10e8eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"
Oct 29 21:55:54.667199 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.666755    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvldb\" (UniqueName: \"kubernetes.io/projected/79e8c570-3e0f-40b7-9696-81b84b10e8eb-kube-api-access-vvldb\") pod \"payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x\" (UID: \"79e8c570-3e0f-40b7-9696-81b84b10e8eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"
Oct 29 21:55:54.667536 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.667491    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e8c570-3e0f-40b7-9696-81b84b10e8eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x\" (UID: \"79e8c570-3e0f-40b7-9696-81b84b10e8eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"
Oct 29 21:55:54.722650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.718613    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvldb\" (UniqueName: \"kubernetes.io/projected/79e8c570-3e0f-40b7-9696-81b84b10e8eb-kube-api-access-vvldb\") pod \"payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x\" (UID: \"79e8c570-3e0f-40b7-9696-81b84b10e8eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"
Oct 29 21:55:54.846243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.846210    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" event={"ID":"9c6e140c-ea25-446f-a3b2-42d49e6634e5","Type":"ContainerStarted","Data":"6810b68ab6cc5c4e5422e7350daec0aa56237daac3fc7651529ddab5dd2f7817"}
Oct 29 21:55:54.846412 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.846264    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" event={"ID":"9c6e140c-ea25-446f-a3b2-42d49e6634e5","Type":"ContainerStarted","Data":"f8b0ffd3297ce9fa7332d1ebc5b587d769a3b2eee31b5f5e571566ca3bc6e2ff"}
Oct 29 21:55:54.856690 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.856670    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"
Oct 29 21:55:54.911652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:54.910777    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" podStartSLOduration=1.91075294 podStartE2EDuration="1.91075294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:54.909442716 +0000 UTC m=+527.409485016" watchObservedRunningTime="2026-10-29 21:55:54.91075294 +0000 UTC m=+527.410795245"
Oct 29 21:55:55.192652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:55.190137    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"]
Oct 29 21:55:55.203662 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:55.201027    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8c570_3e0f_40b7_9696_81b84b10e8eb.slice/crio-a0319ca7d34e60afd0961c975fed38c6a178be012e98c790d5805f2d9f3fc154 WatchSource:0}: Error finding container a0319ca7d34e60afd0961c975fed38c6a178be012e98c790d5805f2d9f3fc154: Status 404 returned error can't find the container with id a0319ca7d34e60afd0961c975fed38c6a178be012e98c790d5805f2d9f3fc154
Oct 29 21:55:55.601438 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:55.601403    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:55.601854 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:55.601831    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:55.603211 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:55.603191    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 21:55:55.630899 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:55.630878    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" path="/var/lib/kubelet/pods/45bb9e0c-17cf-41f8-b79e-841fa270c02b/volumes"
Oct 29 21:55:55.640421 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:55.640397    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:55.646333 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:55.646313    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:55.854465 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:55.854349    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x" event={"ID":"79e8c570-3e0f-40b7-9696-81b84b10e8eb","Type":"ContainerStarted","Data":"e3ab74d6d6c07120f1861fbc0a0b1cea3fe8f7e62a72f6f6ded5d2e329656226"}
Oct 29 21:55:55.854994 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:55.854451    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x" event={"ID":"79e8c570-3e0f-40b7-9696-81b84b10e8eb","Type":"ContainerStarted","Data":"a0319ca7d34e60afd0961c975fed38c6a178be012e98c790d5805f2d9f3fc154"}
Oct 29 21:55:55.854994 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:55.854907    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"
Oct 29 21:55:55.854994 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:55.854956    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:55.855802 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:55.855783    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:55:55.897498 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:55.897454    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x" podStartSLOduration=1.897438466 podStartE2EDuration="1.897438466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:55.894903215 +0000 UTC m=+528.394945516" watchObservedRunningTime="2026-10-29 21:55:55.897438466 +0000 UTC m=+528.397480765"
Oct 29 21:55:57.106358 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.106327    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"]
Oct 29 21:55:57.106969 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.106954    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="45bb9e0c-17cf-41f8-b79e-841fa270c02b" containerName="istio-proxy"
Oct 29 21:55:57.110536 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.110518    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"
Oct 29 21:55:57.124839 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.124815    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-743a5e\""
Oct 29 21:55:57.136459 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.136436    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-743a5e-dockercfg-bqlnk\""
Oct 29 21:55:57.138921 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.138898    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"]
Oct 29 21:55:57.195652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.194827    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"]
Oct 29 21:55:57.209649 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.199615    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b742a630-e75b-4040-8521-12388f3d0427-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk\" (UID: \"b742a630-e75b-4040-8521-12388f3d0427\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"
Oct 29 21:55:57.214832 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.214797    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh7jw\" (UniqueName: \"kubernetes.io/projected/b742a630-e75b-4040-8521-12388f3d0427-kube-api-access-dh7jw\") pod \"payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk\" (UID: \"b742a630-e75b-4040-8521-12388f3d0427\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"
Oct 29 21:55:57.236646 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.236597    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"]
Oct 29 21:55:57.236955 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.236892    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"
Oct 29 21:55:57.315951 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.315914    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b742a630-e75b-4040-8521-12388f3d0427-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk\" (UID: \"b742a630-e75b-4040-8521-12388f3d0427\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"
Oct 29 21:55:57.316115 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.315996    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8bqw\" (UniqueName: \"kubernetes.io/projected/e3a92df7-921d-4f5f-a18b-0c9494dfba37-kube-api-access-j8bqw\") pod \"payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr\" (UID: \"e3a92df7-921d-4f5f-a18b-0c9494dfba37\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"
Oct 29 21:55:57.316177 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.316152    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7jw\" (UniqueName: \"kubernetes.io/projected/b742a630-e75b-4040-8521-12388f3d0427-kube-api-access-dh7jw\") pod \"payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk\" (UID: \"b742a630-e75b-4040-8521-12388f3d0427\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"
Oct 29 21:55:57.316250 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.316226    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a92df7-921d-4f5f-a18b-0c9494dfba37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr\" (UID: \"e3a92df7-921d-4f5f-a18b-0c9494dfba37\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"
Oct 29 21:55:57.316737 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.316719    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b742a630-e75b-4040-8521-12388f3d0427-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk\" (UID: \"b742a630-e75b-4040-8521-12388f3d0427\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"
Oct 29 21:55:57.341679 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.341616    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh7jw\" (UniqueName: \"kubernetes.io/projected/b742a630-e75b-4040-8521-12388f3d0427-kube-api-access-dh7jw\") pod \"payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk\" (UID: \"b742a630-e75b-4040-8521-12388f3d0427\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"
Oct 29 21:55:57.417216 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.417129    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a92df7-921d-4f5f-a18b-0c9494dfba37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr\" (UID: \"e3a92df7-921d-4f5f-a18b-0c9494dfba37\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"
Oct 29 21:55:57.417380 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.417219    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8bqw\" (UniqueName: \"kubernetes.io/projected/e3a92df7-921d-4f5f-a18b-0c9494dfba37-kube-api-access-j8bqw\") pod \"payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr\" (UID: \"e3a92df7-921d-4f5f-a18b-0c9494dfba37\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"
Oct 29 21:55:57.417831 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.417810    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a92df7-921d-4f5f-a18b-0c9494dfba37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr\" (UID: \"e3a92df7-921d-4f5f-a18b-0c9494dfba37\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"
Oct 29 21:55:57.426055 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.426033    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"
Oct 29 21:55:57.443226 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.443201    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8bqw\" (UniqueName: \"kubernetes.io/projected/e3a92df7-921d-4f5f-a18b-0c9494dfba37-kube-api-access-j8bqw\") pod \"payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr\" (UID: \"e3a92df7-921d-4f5f-a18b-0c9494dfba37\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"
Oct 29 21:55:57.562090 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.562052    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"
Oct 29 21:55:57.741670 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.741643    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"]
Oct 29 21:55:57.743259 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:57.743229    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb742a630_e75b_4040_8521_12388f3d0427.slice/crio-9a567e48f4f160a6490945767b0b753d54473348af7fa2c46d2cd72076a814a9 WatchSource:0}: Error finding container 9a567e48f4f160a6490945767b0b753d54473348af7fa2c46d2cd72076a814a9: Status 404 returned error can't find the container with id 9a567e48f4f160a6490945767b0b753d54473348af7fa2c46d2cd72076a814a9
Oct 29 21:55:57.872442 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.872401    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk" event={"ID":"b742a630-e75b-4040-8521-12388f3d0427","Type":"ContainerStarted","Data":"18cc190494e31d0dfacc0b5498f1f7da571342517d08184baeae71bd2d5883d8"}
Oct 29 21:55:57.872598 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.872457    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk" event={"ID":"b742a630-e75b-4040-8521-12388f3d0427","Type":"ContainerStarted","Data":"9a567e48f4f160a6490945767b0b753d54473348af7fa2c46d2cd72076a814a9"}
Oct 29 21:55:57.872692 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.872621    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"
Oct 29 21:55:57.899992 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.899968    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"]
Oct 29 21:55:57.902013 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:55:57.901989    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a92df7_921d_4f5f_a18b_0c9494dfba37.slice/crio-167020242e1f156f5983b9e693e836bc32a1885a21b3368138943503962ff728 WatchSource:0}: Error finding container 167020242e1f156f5983b9e693e836bc32a1885a21b3368138943503962ff728: Status 404 returned error can't find the container with id 167020242e1f156f5983b9e693e836bc32a1885a21b3368138943503962ff728
Oct 29 21:55:57.914361 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:57.914311    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk" podStartSLOduration=0.914294292 podStartE2EDuration="914.294292ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:57.912484944 +0000 UTC m=+530.412527245" watchObservedRunningTime="2026-10-29 21:55:57.914294292 +0000 UTC m=+530.414336593"
Oct 29 21:55:58.411243 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:55:58.411208    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:55:58.420477 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:58.420460    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:58.433433 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:58.433412    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:58.881694 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:58.881653    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr" event={"ID":"e3a92df7-921d-4f5f-a18b-0c9494dfba37","Type":"ContainerStarted","Data":"2182d653f74f40e443dd8498d2133d0df2fbf2720f27132ca808221758b13d17"}
Oct 29 21:55:58.882171 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:58.882137    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"
Oct 29 21:55:58.882329 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:58.882196    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr" event={"ID":"e3a92df7-921d-4f5f-a18b-0c9494dfba37","Type":"ContainerStarted","Data":"167020242e1f156f5983b9e693e836bc32a1885a21b3368138943503962ff728"}
Oct 29 21:55:58.898327 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:58.898303    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:55:58.924973 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:55:58.924933    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr" podStartSLOduration=1.92491997 podStartE2EDuration="1.92491997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:58.923056046 +0000 UTC m=+531.423098346" watchObservedRunningTime="2026-10-29 21:55:58.92491997 +0000 UTC m=+531.424962268"
Oct 29 21:56:01.102532 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:01.102466    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:56:02.840864 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:02.840827    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"
Oct 29 21:56:03.840642 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:03.840596    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"
Oct 29 21:56:06.864774 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:06.864748    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"
Oct 29 21:56:08.886421 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:08.885704    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"
Oct 29 21:56:10.600779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.600739    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 21:56:10.601166 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.600797    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:56:10.787925 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.787893    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"]
Oct 29 21:56:10.800192 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.800160    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:10.835653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.832979    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a940bc\"/\"simulator-w6-a940bc-kserve-self-signed-certs\""
Oct 29 21:56:10.835653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.833163    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a940bc\"/\"kube-root-ca.crt\""
Oct 29 21:56:10.835653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.834056    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a940bc\"/\"default-dockercfg-mcsgh\""
Oct 29 21:56:10.835653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.834174    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a940bc\"/\"openshift-service-ca.crt\""
Oct 29 21:56:10.835653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.834842    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"]
Oct 29 21:56:10.899129 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.899057    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"
Oct 29 21:56:10.989760 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.989727    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tmp-dir\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:10.989921 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.989768    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-model-cache\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:10.989921 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.989853    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-dshm\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:10.989921 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.989910    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-home\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:10.990082 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.989927    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tls-certs\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:10.990148 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:10.990123    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlw5l\" (UniqueName: \"kubernetes.io/projected/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-kube-api-access-qlw5l\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.091755 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.091716    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-dshm\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.092584 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.092556    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-home\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.092760 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.092592    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tls-certs\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.092760 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.092755    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlw5l\" (UniqueName: \"kubernetes.io/projected/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-kube-api-access-qlw5l\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.092881 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.092858    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tmp-dir\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.093040 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.092899    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-model-cache\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.093256 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.093231    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-model-cache\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.093718 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.093653    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tmp-dir\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.094110 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.094086    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-home\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.095297 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.094988    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-dshm\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.096878 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.096850    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tls-certs\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.120313 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.120284    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlw5l\" (UniqueName: \"kubernetes.io/projected/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-kube-api-access-qlw5l\") pod \"simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") " pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.156519 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.156410    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:11.521586 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:11.521554    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"]
Oct 29 21:56:11.523407 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:56:11.523376    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61b0b4e9_d95f_474b_a3f3_3b29bcb723d8.slice/crio-0b97749f66c751b7acf54e229e30a535d2c20c69a3f3cbe57fb8c950db8c7cc2 WatchSource:0}: Error finding container 0b97749f66c751b7acf54e229e30a535d2c20c69a3f3cbe57fb8c950db8c7cc2: Status 404 returned error can't find the container with id 0b97749f66c751b7acf54e229e30a535d2c20c69a3f3cbe57fb8c950db8c7cc2
Oct 29 21:56:12.110804 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:12.110764    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" event={"ID":"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8","Type":"ContainerStarted","Data":"2b5113fa541a6181da648712b000cca709c69325c3a288335ec238cd9014119d"}
Oct 29 21:56:12.111312 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:12.110822    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" event={"ID":"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8","Type":"ContainerStarted","Data":"0b97749f66c751b7acf54e229e30a535d2c20c69a3f3cbe57fb8c950db8c7cc2"}
Oct 29 21:56:12.113568 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:12.113537    7082 generic.go:356] "Generic (PLEG): container finished" podID="84119759-f427-4a8d-960e-8b342fca0d38" containerID="8824abb321ebecc9d637b050ea183c5a3bc42a559dc32b2e6e0254c2320b6290" exitCode=0
Oct 29 21:56:12.113754 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:12.113596    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" event={"ID":"84119759-f427-4a8d-960e-8b342fca0d38","Type":"ContainerDied","Data":"8824abb321ebecc9d637b050ea183c5a3bc42a559dc32b2e6e0254c2320b6290"}
Oct 29 21:56:12.114023 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:12.114008    7082 scope.go:122] "RemoveContainer" containerID="8824abb321ebecc9d637b050ea183c5a3bc42a559dc32b2e6e0254c2320b6290"
Oct 29 21:56:12.115553 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:12.115517    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:56:12.115684 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:12.115609    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:56:12.115747 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:12.115694    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:56:12.154539 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:12.154495    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podStartSLOduration=2.154481932 podStartE2EDuration="2.154481932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:12.153764259 +0000 UTC m=+544.653806558" watchObservedRunningTime="2026-10-29 21:56:12.154481932 +0000 UTC m=+544.654524232"
Oct 29 21:56:12.157964 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:12.157945    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:12.425925 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:12.425893    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:56:13.130975 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:13.130938    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" event={"ID":"84119759-f427-4a8d-960e-8b342fca0d38","Type":"ContainerStarted","Data":"f2ec93756b3a9e6b49a393771834c3f05bb2c5c12e7817d9a7b0a94a48869c8a"}
Oct 29 21:56:13.428536 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:13.428443    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:56:13.428769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:13.428709    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 29 21:56:13.428847 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:13.428758    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 29 21:56:14.429283 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:14.429249    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 29 21:56:14.429909 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:14.429878    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 29 21:56:15.428620 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:15.428579    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 29 21:56:15.428845 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:15.428663    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 29 21:56:16.429325 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:16.429274    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 29 21:56:16.429834 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:16.429348    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 29 21:56:17.428790 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:17.428567    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 29 21:56:17.429067 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:17.429033    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 29 21:56:18.651001 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:18.650970    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:56:18.651478 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:18.651342    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:56:18.652259 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:18.652233    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 21:56:21.157441 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:21.157402    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:21.170016 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:21.169994    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:21.201715 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:21.201692    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:56:21.713322 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:21.713283    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:56:22.247417 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.247378    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"]
Oct 29 21:56:22.253656 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.253612    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.268148 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.268080    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 29 21:56:22.268148 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.268101    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 29 21:56:22.268148 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.268101    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 29 21:56:22.268148 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.268138    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jjwwj\""
Oct 29 21:56:22.282662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.282601    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"]
Oct 29 21:56:22.382647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.379343    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.382647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.379442    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.382647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.379487    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgg9v\" (UniqueName: \"kubernetes.io/projected/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-kube-api-access-kgg9v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.382647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.379516    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.382647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.379576    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.382647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.379652    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.382647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.379741    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.480727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.480696    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.480954 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.480939    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgg9v\" (UniqueName: \"kubernetes.io/projected/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-kube-api-access-kgg9v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.481034 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.481024    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.481105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.481097    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.481192 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.481183    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.481292 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.481283    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.481422 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.481414    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.482506 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.482473    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.494276 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.494213    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.508699 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.494647    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.511159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.510750    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.518956 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.518119    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.551011 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.550975    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.552830 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.552088    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgg9v\" (UniqueName: \"kubernetes.io/projected/1aee5f5c-e3c5-4d61-9082-417a3fd2795d-kube-api-access-kgg9v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8rvj\" (UID: \"1aee5f5c-e3c5-4d61-9082-417a3fd2795d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.567077 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.567051    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:22.956500 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:22.951047    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"]
Oct 29 21:56:22.962761 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:56:22.962337    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aee5f5c_e3c5_4d61_9082_417a3fd2795d.slice/crio-432bb132f41f8526b0d430f216c731427118ccc87af588606a3e60fbc5d63b57 WatchSource:0}: Error finding container 432bb132f41f8526b0d430f216c731427118ccc87af588606a3e60fbc5d63b57: Status 404 returned error can't find the container with id 432bb132f41f8526b0d430f216c731427118ccc87af588606a3e60fbc5d63b57
Oct 29 21:56:23.236003 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:23.235907    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj" event={"ID":"1aee5f5c-e3c5-4d61-9082-417a3fd2795d","Type":"ContainerStarted","Data":"432bb132f41f8526b0d430f216c731427118ccc87af588606a3e60fbc5d63b57"}
Oct 29 21:56:27.481195 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:27.481161    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:56:27.481471 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:27.481226    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:56:28.278820 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:28.278778    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj" event={"ID":"1aee5f5c-e3c5-4d61-9082-417a3fd2795d","Type":"ContainerStarted","Data":"83e3871935544ba30eb467aa25d365058882bd115260e4bdf4d70e21047003be"}
Oct 29 21:56:28.279934 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:28.279909    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:28.282885 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:28.282848    7082 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-g8rvj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 21:56:28.283008 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:28.282910    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj" podUID="1aee5f5c-e3c5-4d61-9082-417a3fd2795d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 21:56:28.355216 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:28.355164    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj" podStartSLOduration=1.8420915450000002 podStartE2EDuration="6.355149443s" totalImagesPullingTime="4.513057898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:28.341566399 +0000 UTC m=+560.841608740" watchObservedRunningTime="2026-10-29 21:56:28.355149443 +0000 UTC m=+560.855191742"
Oct 29 21:56:28.434347 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:56:28.431646    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:56:29.158074 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:29.158032    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:56:29.295709 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:29.291583    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8rvj"
Oct 29 21:56:32.368257 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:32.368216    7082 generic.go:356] "Generic (PLEG): container finished" podID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerID="6810b68ab6cc5c4e5422e7350daec0aa56237daac3fc7651529ddab5dd2f7817" exitCode=0
Oct 29 21:56:32.368829 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:32.368348    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" event={"ID":"9c6e140c-ea25-446f-a3b2-42d49e6634e5","Type":"ContainerDied","Data":"6810b68ab6cc5c4e5422e7350daec0aa56237daac3fc7651529ddab5dd2f7817"}
Oct 29 21:56:32.368944 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:32.368922    7082 scope.go:122] "RemoveContainer" containerID="6810b68ab6cc5c4e5422e7350daec0aa56237daac3fc7651529ddab5dd2f7817"
Oct 29 21:56:32.370930 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:32.370698    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:56:32.370930 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:32.370769    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:56:32.371300 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:32.371072    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:56:33.378362 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:33.378322    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" event={"ID":"9c6e140c-ea25-446f-a3b2-42d49e6634e5","Type":"ContainerStarted","Data":"2faa7e03ec44d32988b2b89c3b88e65a35e3d02a1926e28262215b309747cb4b"}
Oct 29 21:56:33.641106 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:33.641012    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:56:33.641371 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:33.641350    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 21:56:33.641431 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:33.641412    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 21:56:34.641091 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:34.641056    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 21:56:34.641481 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:34.641121    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 21:56:35.641489 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:35.641451    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 21:56:35.641997 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:35.641521    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 21:56:36.641189 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:36.641155    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 21:56:36.641368 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:36.641220    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 21:56:37.640843 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:37.640804    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 29 21:56:37.641346 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:37.640866    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 29 21:56:38.718111 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:38.718075    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:56:38.719658 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:38.719605    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:56:38.720589 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:38.720570    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 21:56:39.292880 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:39.292838    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:56:55.173100 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.173063    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"]
Oct 29 21:56:55.177389 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.177367    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"
Oct 29 21:56:55.194672 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.194418    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 21:56:55.210645 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.207142    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 21:56:55.210645 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.207714    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 21:56:55.210645 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.207788    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 21:56:55.210645 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.210183    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"]
Oct 29 21:56:55.287374 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.287342    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36049559-wll2s\" (UID: \"559e069d-73a9-4764-96ec-a1d3497def06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"
Oct 29 21:56:55.287540 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.287393    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7m4v\" (UniqueName: \"kubernetes.io/projected/559e069d-73a9-4764-96ec-a1d3497def06-kube-api-access-v7m4v\") pod \"maas-api-revoke-sub-36049559-wll2s\" (UID: \"559e069d-73a9-4764-96ec-a1d3497def06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"
Oct 29 21:56:55.287540 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.287492    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36049559-wll2s\" (UID: \"559e069d-73a9-4764-96ec-a1d3497def06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"
Oct 29 21:56:55.389537 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.389496    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36049559-wll2s\" (UID: \"559e069d-73a9-4764-96ec-a1d3497def06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"
Oct 29 21:56:55.389736 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.389579    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7m4v\" (UniqueName: \"kubernetes.io/projected/559e069d-73a9-4764-96ec-a1d3497def06-kube-api-access-v7m4v\") pod \"maas-api-revoke-sub-36049559-wll2s\" (UID: \"559e069d-73a9-4764-96ec-a1d3497def06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"
Oct 29 21:56:55.389736 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.389619    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36049559-wll2s\" (UID: \"559e069d-73a9-4764-96ec-a1d3497def06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"
Oct 29 21:56:55.390328 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.390301    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36049559-wll2s\" (UID: \"559e069d-73a9-4764-96ec-a1d3497def06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"
Oct 29 21:56:55.392233 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.392212    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36049559-wll2s\" (UID: \"559e069d-73a9-4764-96ec-a1d3497def06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"
Oct 29 21:56:55.413422 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.413395    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7m4v\" (UniqueName: \"kubernetes.io/projected/559e069d-73a9-4764-96ec-a1d3497def06-kube-api-access-v7m4v\") pod \"maas-api-revoke-sub-36049559-wll2s\" (UID: \"559e069d-73a9-4764-96ec-a1d3497def06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"
Oct 29 21:56:55.492158 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.492086    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"
Oct 29 21:56:55.806134 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:55.806102    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"]
Oct 29 21:56:55.808761 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:56:55.808732    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559e069d_73a9_4764_96ec_a1d3497def06.slice/crio-7fe18bdce5aa537dd42a839c8dd9a07bd0e9f40cd956ed4a3d510589c1641622 WatchSource:0}: Error finding container 7fe18bdce5aa537dd42a839c8dd9a07bd0e9f40cd956ed4a3d510589c1641622: Status 404 returned error can't find the container with id 7fe18bdce5aa537dd42a839c8dd9a07bd0e9f40cd956ed4a3d510589c1641622
Oct 29 21:56:56.583933 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:56.583900    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s" event={"ID":"559e069d-73a9-4764-96ec-a1d3497def06","Type":"ContainerStarted","Data":"e5acced70ba8d2bdbaaab9508de630f984d8692e62e5cfb919baaf93571454ee"}
Oct 29 21:56:56.584381 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:56.583955    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s" event={"ID":"559e069d-73a9-4764-96ec-a1d3497def06","Type":"ContainerStarted","Data":"7fe18bdce5aa537dd42a839c8dd9a07bd0e9f40cd956ed4a3d510589c1641622"}
Oct 29 21:56:56.628293 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:56.628247    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s" podStartSLOduration=1.62823399 podStartE2EDuration="1.62823399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:56.627455785 +0000 UTC m=+589.127498084" watchObservedRunningTime="2026-10-29 21:56:56.62823399 +0000 UTC m=+589.128276288"
Oct 29 21:56:57.595746 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:57.595710    7082 generic.go:356] "Generic (PLEG): container finished" podID="559e069d-73a9-4764-96ec-a1d3497def06" containerID="e5acced70ba8d2bdbaaab9508de630f984d8692e62e5cfb919baaf93571454ee" exitCode=0
Oct 29 21:56:57.596137 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:57.595774    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s" event={"ID":"559e069d-73a9-4764-96ec-a1d3497def06","Type":"ContainerDied","Data":"e5acced70ba8d2bdbaaab9508de630f984d8692e62e5cfb919baaf93571454ee"}
Oct 29 21:56:58.412680 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:56:58.412641    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:56:58.754641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:58.754598    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"
Oct 29 21:56:58.834253 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:58.834221    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-xks-ca\") pod \"559e069d-73a9-4764-96ec-a1d3497def06\" (UID: \"559e069d-73a9-4764-96ec-a1d3497def06\") "
Oct 29 21:56:58.834429 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:58.834356    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-openshift-ca\") pod \"559e069d-73a9-4764-96ec-a1d3497def06\" (UID: \"559e069d-73a9-4764-96ec-a1d3497def06\") "
Oct 29 21:56:58.834505 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:58.834482    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/559e069d-73a9-4764-96ec-a1d3497def06-kube-api-access-v7m4v\" (UniqueName: \"kubernetes.io/projected/559e069d-73a9-4764-96ec-a1d3497def06-kube-api-access-v7m4v\") pod \"559e069d-73a9-4764-96ec-a1d3497def06\" (UID: \"559e069d-73a9-4764-96ec-a1d3497def06\") "
Oct 29 21:56:58.834761 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:58.834734    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-openshift-ca" pod "559e069d-73a9-4764-96ec-a1d3497def06" (UID: "559e069d-73a9-4764-96ec-a1d3497def06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:56:58.835042 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:58.835020    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:56:58.836520 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:58.836499    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-xks-ca" pod "559e069d-73a9-4764-96ec-a1d3497def06" (UID: "559e069d-73a9-4764-96ec-a1d3497def06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:56:58.836660 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:58.836639    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559e069d-73a9-4764-96ec-a1d3497def06-kube-api-access-v7m4v" pod "559e069d-73a9-4764-96ec-a1d3497def06" (UID: "559e069d-73a9-4764-96ec-a1d3497def06"). InnerVolumeSpecName "kube-api-access-v7m4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:56:58.936210 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:58.936134    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7m4v\" (UniqueName: \"kubernetes.io/projected/559e069d-73a9-4764-96ec-a1d3497def06-kube-api-access-v7m4v\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:56:58.936210 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:58.936163    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559e069d-73a9-4764-96ec-a1d3497def06-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:56:59.541405 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.541368    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"]
Oct 29 21:56:59.542179 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.542158    7082 cpu_manager.go:414] "containerMap: removing container" podUID="559e069d-73a9-4764-96ec-a1d3497def06" containerName="revoke-keys"
Oct 29 21:56:59.542179 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.542179    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="559e069d-73a9-4764-96ec-a1d3497def06" containerName="revoke-keys"
Oct 29 21:56:59.542305 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.542270    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="559e069d-73a9-4764-96ec-a1d3497def06" containerName="revoke-keys"
Oct 29 21:56:59.545982 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.545964    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.561416 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.561394    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b31f3f-openshift-default-dockercfg-ddh7z\""
Oct 29 21:56:59.575454 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.575430    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"]
Oct 29 21:56:59.620992 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.620954    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s" event={"ID":"559e069d-73a9-4764-96ec-a1d3497def06","Type":"ContainerDied","Data":"7fe18bdce5aa537dd42a839c8dd9a07bd0e9f40cd956ed4a3d510589c1641622"}
Oct 29 21:56:59.621273 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.621257    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fe18bdce5aa537dd42a839c8dd9a07bd0e9f40cd956ed4a3d510589c1641622"
Oct 29 21:56:59.621486 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.621433    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"
Oct 29 21:56:59.644740 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.644709    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.644847 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.644760    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.644906 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.644846    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.644942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.644913    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.645029 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.644947    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.645029 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.644978    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgl5b\" (UniqueName: \"kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-kube-api-access-qgl5b\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.645108 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.645056    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.645108 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.645081    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.645187 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.645115    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.746725 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.746689    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.746892 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.746748    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.746892 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.746803    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.746892 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.746851    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.746892 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.746883    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.747082 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.747049    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgl5b\" (UniqueName: \"kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-kube-api-access-qgl5b\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.747136 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.747122    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.747189 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.747152    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.747246 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.747194    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.747246 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.747216    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.747246 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.747218    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.747410 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.747250    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.747491 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.747472    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.747619 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.747600    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.749513 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.749491    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.749744 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.749721    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.770331 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.770304    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.770610 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.770380    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgl5b\" (UniqueName: \"kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-kube-api-access-qgl5b\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:56:59.859792 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:56:59.859761    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:57:00.196559 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:00.196526    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"]
Oct 29 21:57:00.197070 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:57:00.197039    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56d6c21_ed20_4380_b2f5_69b9d168f5c5.slice/crio-6c83f432e75164f44f15ecf407d534fa9bfbd8d6a0e8ef0ea80ce8ce8b9e3dc4 WatchSource:0}: Error finding container 6c83f432e75164f44f15ecf407d534fa9bfbd8d6a0e8ef0ea80ce8ce8b9e3dc4: Status 404 returned error can't find the container with id 6c83f432e75164f44f15ecf407d534fa9bfbd8d6a0e8ef0ea80ce8ce8b9e3dc4
Oct 29 21:57:00.199743 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:00.199707    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:57:00.199860 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:00.199777    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:57:00.199860 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:00.199821    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:57:00.632210 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:00.632163    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" event={"ID":"d56d6c21-ed20-4380-b2f5-69b9d168f5c5","Type":"ContainerStarted","Data":"19713b4347b065627314564bc48679a01f81b7613b28b3ba6564fce40621b315"}
Oct 29 21:57:00.632428 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:00.632221    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" event={"ID":"d56d6c21-ed20-4380-b2f5-69b9d168f5c5","Type":"ContainerStarted","Data":"6c83f432e75164f44f15ecf407d534fa9bfbd8d6a0e8ef0ea80ce8ce8b9e3dc4"}
Oct 29 21:57:00.682373 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:00.682310    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" podStartSLOduration=1.682294599 podStartE2EDuration="1.682294599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:00.680009528 +0000 UTC m=+593.180051830" watchObservedRunningTime="2026-10-29 21:57:00.682294599 +0000 UTC m=+593.182336898"
Oct 29 21:57:00.860249 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:00.860215    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:57:01.863357 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:01.863140    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 29 21:57:01.863357 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:01.863228    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" podUID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 29 21:57:02.860712 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:02.860672    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 29 21:57:02.860906 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:02.860740    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" podUID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 29 21:57:03.860749 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:03.860698    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 29 21:57:03.861233 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:03.860779    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" podUID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 29 21:57:04.860614 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:04.860578    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 29 21:57:04.860839 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:04.860686    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" podUID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 29 21:57:05.860590 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:05.860556    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 29 21:57:05.860796 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:05.860612    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" podUID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 29 21:57:06.865092 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:06.865065    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:57:06.865581 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:06.865563    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:57:06.866286 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:06.866270    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 21:57:07.288931 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:07.288836    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:57:09.899764 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:09.899728    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"]
Oct 29 21:57:09.908247 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:09.908218    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"
Oct 29 21:57:09.923486 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:09.923441    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"]
Oct 29 21:57:09.935754 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:09.931843    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 21:57:09.935754 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:09.931896    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 21:57:09.935754 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:09.931975    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 21:57:09.935754 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:09.932014    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 21:57:10.111237 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:10.111178    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mpdq\" (UniqueName: \"kubernetes.io/projected/1b2dfa1e-1c25-45b8-8127-4c95f1657553-kube-api-access-2mpdq\") pod \"maas-api-revoke-sub-f65659a3-zc4sm\" (UID: \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"
Oct 29 21:57:10.112154 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:10.112128    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f65659a3-zc4sm\" (UID: \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"
Oct 29 21:57:10.112874 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:10.112854    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f65659a3-zc4sm\" (UID: \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"
Oct 29 21:57:10.216284 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:10.216197    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mpdq\" (UniqueName: \"kubernetes.io/projected/1b2dfa1e-1c25-45b8-8127-4c95f1657553-kube-api-access-2mpdq\") pod \"maas-api-revoke-sub-f65659a3-zc4sm\" (UID: \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"
Oct 29 21:57:10.220493 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:10.220462    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f65659a3-zc4sm\" (UID: \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"
Oct 29 21:57:10.221740 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:10.221720    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f65659a3-zc4sm\" (UID: \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"
Oct 29 21:57:10.222441 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:10.221536    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f65659a3-zc4sm\" (UID: \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"
Oct 29 21:57:10.270446 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:10.264202    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f65659a3-zc4sm\" (UID: \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"
Oct 29 21:57:10.270446 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:10.265216    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mpdq\" (UniqueName: \"kubernetes.io/projected/1b2dfa1e-1c25-45b8-8127-4c95f1657553-kube-api-access-2mpdq\") pod \"maas-api-revoke-sub-f65659a3-zc4sm\" (UID: \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"
Oct 29 21:57:10.534705 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:10.532825    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"
Oct 29 21:57:10.932980 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:10.932871    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"]
Oct 29 21:57:10.939651 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:57:10.935925    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2dfa1e_1c25_45b8_8127_4c95f1657553.slice/crio-c627e5e7b3a06f027333ee02a3964f7c5d3a21467c1d996673d19c9599709bfb WatchSource:0}: Error finding container c627e5e7b3a06f027333ee02a3964f7c5d3a21467c1d996673d19c9599709bfb: Status 404 returned error can't find the container with id c627e5e7b3a06f027333ee02a3964f7c5d3a21467c1d996673d19c9599709bfb
Oct 29 21:57:11.759084 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:11.759046    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm" event={"ID":"1b2dfa1e-1c25-45b8-8127-4c95f1657553","Type":"ContainerStarted","Data":"4dad66ae77a60ce438a2bf2495c19cc90e5e15e30cd1e7a3643059076453b0e7"}
Oct 29 21:57:11.759272 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:11.759119    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm" event={"ID":"1b2dfa1e-1c25-45b8-8127-4c95f1657553","Type":"ContainerStarted","Data":"c627e5e7b3a06f027333ee02a3964f7c5d3a21467c1d996673d19c9599709bfb"}
Oct 29 21:57:11.801403 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:11.801335    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm" podStartSLOduration=2.801315389 podStartE2EDuration="2.801315389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:11.800378837 +0000 UTC m=+604.300421137" watchObservedRunningTime="2026-10-29 21:57:11.801315389 +0000 UTC m=+604.301357689"
Oct 29 21:57:12.768141 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:12.768104    7082 generic.go:356] "Generic (PLEG): container finished" podID="1b2dfa1e-1c25-45b8-8127-4c95f1657553" containerID="4dad66ae77a60ce438a2bf2495c19cc90e5e15e30cd1e7a3643059076453b0e7" exitCode=0
Oct 29 21:57:12.768541 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:12.768198    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm" event={"ID":"1b2dfa1e-1c25-45b8-8127-4c95f1657553","Type":"ContainerDied","Data":"4dad66ae77a60ce438a2bf2495c19cc90e5e15e30cd1e7a3643059076453b0e7"}
Oct 29 21:57:13.919576 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:13.919547    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"
Oct 29 21:57:13.990804 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:13.990772    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-xks-ca\") pod \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\" (UID: \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\") "
Oct 29 21:57:13.991013 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:13.990997    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-openshift-ca\") pod \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\" (UID: \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\") "
Oct 29 21:57:13.991076 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:13.991033    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2dfa1e-1c25-45b8-8127-4c95f1657553-kube-api-access-2mpdq\" (UniqueName: \"kubernetes.io/projected/1b2dfa1e-1c25-45b8-8127-4c95f1657553-kube-api-access-2mpdq\") pod \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\" (UID: \"1b2dfa1e-1c25-45b8-8127-4c95f1657553\") "
Oct 29 21:57:13.991439 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:13.991404    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-openshift-ca" pod "1b2dfa1e-1c25-45b8-8127-4c95f1657553" (UID: "1b2dfa1e-1c25-45b8-8127-4c95f1657553"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:57:13.991775 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:13.991751    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:13.994084 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:13.994058    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2dfa1e-1c25-45b8-8127-4c95f1657553-kube-api-access-2mpdq" pod "1b2dfa1e-1c25-45b8-8127-4c95f1657553" (UID: "1b2dfa1e-1c25-45b8-8127-4c95f1657553"). InnerVolumeSpecName "kube-api-access-2mpdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:57:13.994084 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:13.994072    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-xks-ca" pod "1b2dfa1e-1c25-45b8-8127-4c95f1657553" (UID: "1b2dfa1e-1c25-45b8-8127-4c95f1657553"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:57:14.093303 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:14.093268    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mpdq\" (UniqueName: \"kubernetes.io/projected/1b2dfa1e-1c25-45b8-8127-4c95f1657553-kube-api-access-2mpdq\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:14.093475 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:14.093308    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2dfa1e-1c25-45b8-8127-4c95f1657553-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:14.783680 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:14.783653    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"
Oct 29 21:57:14.783875 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:14.783649    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm" event={"ID":"1b2dfa1e-1c25-45b8-8127-4c95f1657553","Type":"ContainerDied","Data":"c627e5e7b3a06f027333ee02a3964f7c5d3a21467c1d996673d19c9599709bfb"}
Oct 29 21:57:14.783875 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:14.783772    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c627e5e7b3a06f027333ee02a3964f7c5d3a21467c1d996673d19c9599709bfb"
Oct 29 21:57:22.375892 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.375859    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"]
Oct 29 21:57:22.376616 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.376594    7082 cpu_manager.go:414] "containerMap: removing container" podUID="1b2dfa1e-1c25-45b8-8127-4c95f1657553" containerName="revoke-keys"
Oct 29 21:57:22.376616 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.376618    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2dfa1e-1c25-45b8-8127-4c95f1657553" containerName="revoke-keys"
Oct 29 21:57:22.376852 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.376836    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2dfa1e-1c25-45b8-8127-4c95f1657553" containerName="revoke-keys"
Oct 29 21:57:22.380185 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.380164    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"
Oct 29 21:57:22.396898 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.396850    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 21:57:22.396898 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.396853    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 21:57:22.408477 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.408454    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 21:57:22.408616 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.408483    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 21:57:22.411361 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.411339    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"]
Oct 29 21:57:22.494452 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.494416    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5055b163-twtsr\" (UID: \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"
Oct 29 21:57:22.494606 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.494574    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpx8j\" (UniqueName: \"kubernetes.io/projected/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-kube-api-access-bpx8j\") pod \"maas-api-revoke-sub-5055b163-twtsr\" (UID: \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"
Oct 29 21:57:22.494677 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.494607    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5055b163-twtsr\" (UID: \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"
Oct 29 21:57:22.595717 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.595685    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5055b163-twtsr\" (UID: \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"
Oct 29 21:57:22.595852 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.595835    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpx8j\" (UniqueName: \"kubernetes.io/projected/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-kube-api-access-bpx8j\") pod \"maas-api-revoke-sub-5055b163-twtsr\" (UID: \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"
Oct 29 21:57:22.595888 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.595866    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5055b163-twtsr\" (UID: \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"
Oct 29 21:57:22.596399 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.596379    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5055b163-twtsr\" (UID: \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"
Oct 29 21:57:22.598332 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.598308    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5055b163-twtsr\" (UID: \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"
Oct 29 21:57:22.619006 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.618978    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpx8j\" (UniqueName: \"kubernetes.io/projected/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-kube-api-access-bpx8j\") pod \"maas-api-revoke-sub-5055b163-twtsr\" (UID: \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"
Oct 29 21:57:22.694392 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:22.694298    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"
Oct 29 21:57:23.004997 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:23.004964    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"]
Oct 29 21:57:23.006072 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:57:23.006044    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4d4672_2bd0_41fb_8bff_f0cfb2469e08.slice/crio-68ab4adc73262d863ed9c26e5c2e07eff4be7891d73b8e9ab88bc1d4fdd4cd9a WatchSource:0}: Error finding container 68ab4adc73262d863ed9c26e5c2e07eff4be7891d73b8e9ab88bc1d4fdd4cd9a: Status 404 returned error can't find the container with id 68ab4adc73262d863ed9c26e5c2e07eff4be7891d73b8e9ab88bc1d4fdd4cd9a
Oct 29 21:57:23.850654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:23.850590    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr" event={"ID":"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08","Type":"ContainerStarted","Data":"3a627afe5dc21c6e67d963b7644be36e05fceb458400de57a692a8ba04545a40"}
Oct 29 21:57:23.851033 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:23.850678    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr" event={"ID":"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08","Type":"ContainerStarted","Data":"68ab4adc73262d863ed9c26e5c2e07eff4be7891d73b8e9ab88bc1d4fdd4cd9a"}
Oct 29 21:57:23.894457 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:23.894394    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr" podStartSLOduration=1.89437558 podStartE2EDuration="1.89437558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:23.891996133 +0000 UTC m=+616.392038434" watchObservedRunningTime="2026-10-29 21:57:23.89437558 +0000 UTC m=+616.394417880"
Oct 29 21:57:24.858915 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:24.858874    7082 generic.go:356] "Generic (PLEG): container finished" podID="5e4d4672-2bd0-41fb-8bff-f0cfb2469e08" containerID="3a627afe5dc21c6e67d963b7644be36e05fceb458400de57a692a8ba04545a40" exitCode=0
Oct 29 21:57:24.859335 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:24.858920    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr" event={"ID":"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08","Type":"ContainerDied","Data":"3a627afe5dc21c6e67d963b7644be36e05fceb458400de57a692a8ba04545a40"}
Oct 29 21:57:26.021474 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:26.021438    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"
Oct 29 21:57:26.148521 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:26.148440    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-xks-ca\") pod \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\" (UID: \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\") "
Oct 29 21:57:26.148737 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:26.148556    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-openshift-ca\") pod \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\" (UID: \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\") "
Oct 29 21:57:26.148737 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:26.148693    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-kube-api-access-bpx8j\" (UniqueName: \"kubernetes.io/projected/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-kube-api-access-bpx8j\") pod \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\" (UID: \"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08\") "
Oct 29 21:57:26.148995 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:26.148967    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-openshift-ca" pod "5e4d4672-2bd0-41fb-8bff-f0cfb2469e08" (UID: "5e4d4672-2bd0-41fb-8bff-f0cfb2469e08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:57:26.149154 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:26.149135    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:26.151079 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:26.151049    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-kube-api-access-bpx8j" pod "5e4d4672-2bd0-41fb-8bff-f0cfb2469e08" (UID: "5e4d4672-2bd0-41fb-8bff-f0cfb2469e08"). InnerVolumeSpecName "kube-api-access-bpx8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:57:26.151172 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:26.151127    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-xks-ca" pod "5e4d4672-2bd0-41fb-8bff-f0cfb2469e08" (UID: "5e4d4672-2bd0-41fb-8bff-f0cfb2469e08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:57:26.250192 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:26.250159    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:26.250192 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:26.250189    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpx8j\" (UniqueName: \"kubernetes.io/projected/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08-kube-api-access-bpx8j\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:26.876848 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:26.876820    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"
Oct 29 21:57:26.876848 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:26.876837    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr" event={"ID":"5e4d4672-2bd0-41fb-8bff-f0cfb2469e08","Type":"ContainerDied","Data":"68ab4adc73262d863ed9c26e5c2e07eff4be7891d73b8e9ab88bc1d4fdd4cd9a"}
Oct 29 21:57:26.877125 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:26.876896    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68ab4adc73262d863ed9c26e5c2e07eff4be7891d73b8e9ab88bc1d4fdd4cd9a"
Oct 29 21:57:28.414208 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:57:28.414168    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:57:31.503468 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.503164    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"]
Oct 29 21:57:31.504519 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.504496    7082 cpu_manager.go:414] "containerMap: removing container" podUID="5e4d4672-2bd0-41fb-8bff-f0cfb2469e08" containerName="revoke-keys"
Oct 29 21:57:31.504519 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.504516    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4d4672-2bd0-41fb-8bff-f0cfb2469e08" containerName="revoke-keys"
Oct 29 21:57:31.504800 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.504763    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4d4672-2bd0-41fb-8bff-f0cfb2469e08" containerName="revoke-keys"
Oct 29 21:57:31.508928 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.508906    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.536743 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.536711    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"]
Oct 29 21:57:31.614200 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.614156    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.614399 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.614215    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llrnf\" (UniqueName: \"kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-kube-api-access-llrnf\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.614399 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.614382    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.614508 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.614432    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.614543 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.614512    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.614588 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.614545    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.614700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.614683    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.614767 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.614726    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.614849 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.614832    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.716191 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.716153    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.716353 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.716209    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.716353 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.716256    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.716353 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.716288    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.716506 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.716352    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.716506 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.716383    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.716506 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.716430    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.716691 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.716551    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.716691 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.716580    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llrnf\" (UniqueName: \"kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-kube-api-access-llrnf\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.716691 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.716599    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.716840 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.716735    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.716840 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.716771    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.716970 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.716940    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.717441 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.717417    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.719797 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.719770    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.719890 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.719871    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.740334 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.740308    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.740435 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.740342    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llrnf\" (UniqueName: \"kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-kube-api-access-llrnf\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:31.823088 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:31.823055    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:32.183192 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:32.183163    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"]
Oct 29 21:57:32.184255 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:57:32.184225    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda251aea6_7480_407f_8e4e_2e3d3c5c99b4.slice/crio-2d210a3f414e3e7fa518b67665efd773d856685dca49791f578c9508f9984720 WatchSource:0}: Error finding container 2d210a3f414e3e7fa518b67665efd773d856685dca49791f578c9508f9984720: Status 404 returned error can't find the container with id 2d210a3f414e3e7fa518b67665efd773d856685dca49791f578c9508f9984720
Oct 29 21:57:32.187595 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:32.187560    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:57:32.187742 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:32.187672    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:57:32.187742 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:32.187727    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:57:32.927866 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:32.927823    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" event={"ID":"a251aea6-7480-407f-8e4e-2e3d3c5c99b4","Type":"ContainerStarted","Data":"e2beb3fe2a4e75249e619eaa7eb02c810f660add66516c4b0265e3ba86605de3"}
Oct 29 21:57:32.928384 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:32.927892    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" event={"ID":"a251aea6-7480-407f-8e4e-2e3d3c5c99b4","Type":"ContainerStarted","Data":"2d210a3f414e3e7fa518b67665efd773d856685dca49791f578c9508f9984720"}
Oct 29 21:57:33.002171 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:33.002104    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" podStartSLOduration=2.002081579 podStartE2EDuration="2.002081579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:32.990008284 +0000 UTC m=+625.490050596" watchObservedRunningTime="2026-10-29 21:57:33.002081579 +0000 UTC m=+625.502123880"
Oct 29 21:57:33.823322 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:33.823276    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:33.825026 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:33.824995    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:57:33.825173 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:33.825047    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 29 21:57:34.823783 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:34.823742    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:57:34.824281 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:34.823807    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 29 21:57:35.101705 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:35.100793    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:57:35.823914 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:35.823864    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:57:35.824324 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:35.823948    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 29 21:57:36.826680 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:36.824003    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:57:36.826680 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:36.824090    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 29 21:57:37.828811 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:37.828775    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:57:37.829389 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:37.829357    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 29 21:57:38.824355 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:38.824318    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:57:38.824991 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:38.824872    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 29 21:57:39.987151 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:39.987111    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:39.989546 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:39.989527    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:39.991799 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:39.991783    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 21:57:40.794380 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:40.794332    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"]
Oct 29 21:57:40.801892 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:40.801862    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"
Oct 29 21:57:40.817486 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:40.817456    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"]
Oct 29 21:57:40.855673 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:40.855342    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 21:57:40.855673 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:40.855351    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 21:57:40.855673 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:40.855347    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 21:57:40.855673 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:40.855412    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 21:57:40.900188 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:40.900152    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78774571-5q2hc\" (UID: \"079f1f32-0b82-4c00-90fa-3eddd0260866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"
Oct 29 21:57:40.900188 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:40.900201    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22bg5\" (UniqueName: \"kubernetes.io/projected/079f1f32-0b82-4c00-90fa-3eddd0260866-kube-api-access-22bg5\") pod \"maas-api-revoke-sub-78774571-5q2hc\" (UID: \"079f1f32-0b82-4c00-90fa-3eddd0260866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"
Oct 29 21:57:40.900415 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:40.900325    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78774571-5q2hc\" (UID: \"079f1f32-0b82-4c00-90fa-3eddd0260866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"
Oct 29 21:57:41.009648 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:41.001975    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78774571-5q2hc\" (UID: \"079f1f32-0b82-4c00-90fa-3eddd0260866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"
Oct 29 21:57:41.009648 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:41.002030    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22bg5\" (UniqueName: \"kubernetes.io/projected/079f1f32-0b82-4c00-90fa-3eddd0260866-kube-api-access-22bg5\") pod \"maas-api-revoke-sub-78774571-5q2hc\" (UID: \"079f1f32-0b82-4c00-90fa-3eddd0260866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"
Oct 29 21:57:41.009648 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:41.002100    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78774571-5q2hc\" (UID: \"079f1f32-0b82-4c00-90fa-3eddd0260866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"
Oct 29 21:57:41.009648 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:41.003196    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78774571-5q2hc\" (UID: \"079f1f32-0b82-4c00-90fa-3eddd0260866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"
Oct 29 21:57:41.009648 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:41.005533    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78774571-5q2hc\" (UID: \"079f1f32-0b82-4c00-90fa-3eddd0260866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"
Oct 29 21:57:41.027541 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:41.027496    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22bg5\" (UniqueName: \"kubernetes.io/projected/079f1f32-0b82-4c00-90fa-3eddd0260866-kube-api-access-22bg5\") pod \"maas-api-revoke-sub-78774571-5q2hc\" (UID: \"079f1f32-0b82-4c00-90fa-3eddd0260866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"
Oct 29 21:57:41.121370 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:41.121329    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"
Oct 29 21:57:41.432469 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:41.432434    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"]
Oct 29 21:57:41.433853 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:57:41.433822    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079f1f32_0b82_4c00_90fa_3eddd0260866.slice/crio-b81f9b683d94f1ef42a0f94ba1f268cdddc8e815f03816b5e5617fefa118a1bb WatchSource:0}: Error finding container b81f9b683d94f1ef42a0f94ba1f268cdddc8e815f03816b5e5617fefa118a1bb: Status 404 returned error can't find the container with id b81f9b683d94f1ef42a0f94ba1f268cdddc8e815f03816b5e5617fefa118a1bb
Oct 29 21:57:42.055014 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:42.054979    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc" event={"ID":"079f1f32-0b82-4c00-90fa-3eddd0260866","Type":"ContainerStarted","Data":"3cfd18bbbb255d6d36b086fc1475062ce6cb40d9860d7bcae9fd4228a1207a7c"}
Oct 29 21:57:42.055484 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:42.055039    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc" event={"ID":"079f1f32-0b82-4c00-90fa-3eddd0260866","Type":"ContainerStarted","Data":"b81f9b683d94f1ef42a0f94ba1f268cdddc8e815f03816b5e5617fefa118a1bb"}
Oct 29 21:57:42.100318 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:42.100267    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc" podStartSLOduration=2.100252696 podStartE2EDuration="2.100252696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:42.098173 +0000 UTC m=+634.598215302" watchObservedRunningTime="2026-10-29 21:57:42.100252696 +0000 UTC m=+634.600294996"
Oct 29 21:57:43.064079 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:43.064032    7082 generic.go:356] "Generic (PLEG): container finished" podID="079f1f32-0b82-4c00-90fa-3eddd0260866" containerID="3cfd18bbbb255d6d36b086fc1475062ce6cb40d9860d7bcae9fd4228a1207a7c" exitCode=0
Oct 29 21:57:43.064486 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:43.064113    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc" event={"ID":"079f1f32-0b82-4c00-90fa-3eddd0260866","Type":"ContainerDied","Data":"3cfd18bbbb255d6d36b086fc1475062ce6cb40d9860d7bcae9fd4228a1207a7c"}
Oct 29 21:57:44.214933 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:44.214902    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"
Oct 29 21:57:44.243518 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:44.243479    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079f1f32-0b82-4c00-90fa-3eddd0260866-kube-api-access-22bg5\" (UniqueName: \"kubernetes.io/projected/079f1f32-0b82-4c00-90fa-3eddd0260866-kube-api-access-22bg5\") pod \"079f1f32-0b82-4c00-90fa-3eddd0260866\" (UID: \"079f1f32-0b82-4c00-90fa-3eddd0260866\") "
Oct 29 21:57:44.243675 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:44.243653    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-xks-ca\") pod \"079f1f32-0b82-4c00-90fa-3eddd0260866\" (UID: \"079f1f32-0b82-4c00-90fa-3eddd0260866\") "
Oct 29 21:57:44.243789 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:44.243764    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-openshift-ca\") pod \"079f1f32-0b82-4c00-90fa-3eddd0260866\" (UID: \"079f1f32-0b82-4c00-90fa-3eddd0260866\") "
Oct 29 21:57:44.244109 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:44.244086    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-openshift-ca" pod "079f1f32-0b82-4c00-90fa-3eddd0260866" (UID: "079f1f32-0b82-4c00-90fa-3eddd0260866"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:57:44.244311 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:44.244295    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:44.245795 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:44.245765    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079f1f32-0b82-4c00-90fa-3eddd0260866-kube-api-access-22bg5" pod "079f1f32-0b82-4c00-90fa-3eddd0260866" (UID: "079f1f32-0b82-4c00-90fa-3eddd0260866"). InnerVolumeSpecName "kube-api-access-22bg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:57:44.245879 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:44.245792    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-xks-ca" pod "079f1f32-0b82-4c00-90fa-3eddd0260866" (UID: "079f1f32-0b82-4c00-90fa-3eddd0260866"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:57:44.346114 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:44.346025    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22bg5\" (UniqueName: \"kubernetes.io/projected/079f1f32-0b82-4c00-90fa-3eddd0260866-kube-api-access-22bg5\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:44.346114 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:44.346059    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/079f1f32-0b82-4c00-90fa-3eddd0260866-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:45.088868 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:45.088818    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc" event={"ID":"079f1f32-0b82-4c00-90fa-3eddd0260866","Type":"ContainerDied","Data":"b81f9b683d94f1ef42a0f94ba1f268cdddc8e815f03816b5e5617fefa118a1bb"}
Oct 29 21:57:45.089046 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:45.088902    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b81f9b683d94f1ef42a0f94ba1f268cdddc8e815f03816b5e5617fefa118a1bb"
Oct 29 21:57:45.089046 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:45.088962    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"
Oct 29 21:57:48.750656 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.750554    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"]
Oct 29 21:57:48.751174 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.751155    7082 cpu_manager.go:414] "containerMap: removing container" podUID="079f1f32-0b82-4c00-90fa-3eddd0260866" containerName="revoke-keys"
Oct 29 21:57:48.751234 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.751178    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079f1f32-0b82-4c00-90fa-3eddd0260866" containerName="revoke-keys"
Oct 29 21:57:48.751413 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.751393    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="079f1f32-0b82-4c00-90fa-3eddd0260866" containerName="revoke-keys"
Oct 29 21:57:48.755512 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.755494    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.772161 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.772136    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a9def7-openshift-default-dockercfg-qflmj\""
Oct 29 21:57:48.788618 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.788591    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"]
Oct 29 21:57:48.812593 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.812550    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.812976 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.812606    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.812976 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.812655    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.812976 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.812741    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmk7m\" (UniqueName: \"kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-kube-api-access-kmk7m\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.812976 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.812803    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.812976 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.812850    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.812976 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.812950    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.842964 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.838885    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.842964 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.839010    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.940217 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.940184    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.940217 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.940222    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.940432 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.940244    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.940432 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.940289    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmk7m\" (UniqueName: \"kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-kube-api-access-kmk7m\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.940432 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.940323    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.940432 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.940353    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.940670 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.940442    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.940670 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.940480    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.940670 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.940514    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.940839 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.940664    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.940839 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.940683    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.940839 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.940787    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.940999 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.940911    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.941167 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.941145    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.943085 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.943060    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.943176 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.943134    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.963739 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.963717    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:48.963938 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:48.963915    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmk7m\" (UniqueName: \"kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-kube-api-access-kmk7m\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:49.068779 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:49.068745    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:49.415454 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:49.415427    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"]
Oct 29 21:57:49.416882 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:57:49.416853    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8cbca6_8215_4ea1_b10a_4e81fa8020d9.slice/crio-eadebf8bdd4a5943b18d0343d8ce2135ec5ed76eb2bde4a8ec4a46fafe6fb18b WatchSource:0}: Error finding container eadebf8bdd4a5943b18d0343d8ce2135ec5ed76eb2bde4a8ec4a46fafe6fb18b: Status 404 returned error can't find the container with id eadebf8bdd4a5943b18d0343d8ce2135ec5ed76eb2bde4a8ec4a46fafe6fb18b
Oct 29 21:57:49.419667 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:49.419607    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:57:49.419757 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:49.419702    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:57:49.419757 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:49.419731    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:57:50.075687 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.075651    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"]
Oct 29 21:57:50.078846 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.078827    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"
Oct 29 21:57:50.093445 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.093421    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 21:57:50.105207 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.105186    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 21:57:50.105360 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.105192    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 21:57:50.105469 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.105240    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 21:57:50.110284 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.110249    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"]
Oct 29 21:57:50.141027 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.140993    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" event={"ID":"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9","Type":"ContainerStarted","Data":"7aad63497c30527e83474c40b35e6cbba76cac0a29d3eaa5bf4b095eee43dd27"}
Oct 29 21:57:50.141167 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.141056    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" event={"ID":"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9","Type":"ContainerStarted","Data":"eadebf8bdd4a5943b18d0343d8ce2135ec5ed76eb2bde4a8ec4a46fafe6fb18b"}
Oct 29 21:57:50.160014 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.159985    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f840c335-rszds\" (UID: \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"
Oct 29 21:57:50.160150 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.160121    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f840c335-rszds\" (UID: \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"
Oct 29 21:57:50.160437 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.160406    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nj29\" (UniqueName: \"kubernetes.io/projected/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-kube-api-access-5nj29\") pod \"maas-api-revoke-sub-f840c335-rszds\" (UID: \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"
Oct 29 21:57:50.227885 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.227831    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" podStartSLOduration=2.22781599 podStartE2EDuration="2.22781599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:50.227267537 +0000 UTC m=+642.727309837" watchObservedRunningTime="2026-10-29 21:57:50.22781599 +0000 UTC m=+642.727858288"
Oct 29 21:57:50.261674 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.261569    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nj29\" (UniqueName: \"kubernetes.io/projected/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-kube-api-access-5nj29\") pod \"maas-api-revoke-sub-f840c335-rszds\" (UID: \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"
Oct 29 21:57:50.261857 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.261702    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f840c335-rszds\" (UID: \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"
Oct 29 21:57:50.261857 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.261777    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f840c335-rszds\" (UID: \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"
Oct 29 21:57:50.262434 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.262410    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f840c335-rszds\" (UID: \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"
Oct 29 21:57:50.264210 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.264187    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f840c335-rszds\" (UID: \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"
Oct 29 21:57:50.316637 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.316592    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nj29\" (UniqueName: \"kubernetes.io/projected/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-kube-api-access-5nj29\") pod \"maas-api-revoke-sub-f840c335-rszds\" (UID: \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"
Oct 29 21:57:50.405869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.405768    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"
Oct 29 21:57:50.775943 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:50.775917    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"]
Oct 29 21:57:50.777676 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:57:50.777647    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5bbe37_aae9_4c32_96a0_dbc09a566fd6.slice/crio-2f6f65f576da998660e38e75f26bed4df4dc8fda9849a3685d577681af8e38e4 WatchSource:0}: Error finding container 2f6f65f576da998660e38e75f26bed4df4dc8fda9849a3685d577681af8e38e4: Status 404 returned error can't find the container with id 2f6f65f576da998660e38e75f26bed4df4dc8fda9849a3685d577681af8e38e4
Oct 29 21:57:51.069611 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.069580    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:51.071307 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.071281    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:57:51.071421 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.071330    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 21:57:51.149128 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.149096    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds" event={"ID":"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6","Type":"ContainerStarted","Data":"85cbffe183e21c21296dba518043aea9c36fc408cf5f49b20dba4fcbf779c785"}
Oct 29 21:57:51.149555 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.149152    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds" event={"ID":"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6","Type":"ContainerStarted","Data":"2f6f65f576da998660e38e75f26bed4df4dc8fda9849a3685d577681af8e38e4"}
Oct 29 21:57:51.219652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.209824    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds" podStartSLOduration=1.209804218 podStartE2EDuration="1.209804218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:51.208580593 +0000 UTC m=+643.708622906" watchObservedRunningTime="2026-10-29 21:57:51.209804218 +0000 UTC m=+643.709846518"
Oct 29 21:57:51.232190 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.230075    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"]
Oct 29 21:57:51.233500 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.233475    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"]
Oct 29 21:57:51.233651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.233585    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"
Oct 29 21:57:51.274652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.274157    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47hjh\" (UniqueName: \"kubernetes.io/projected/b091f9da-2dd9-466b-b7b1-31b685b71481-kube-api-access-47hjh\") pod \"maas-api-revoke-sub-b7f7a1cf-25dlm\" (UID: \"b091f9da-2dd9-466b-b7b1-31b685b71481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"
Oct 29 21:57:51.274652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.274393    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7f7a1cf-25dlm\" (UID: \"b091f9da-2dd9-466b-b7b1-31b685b71481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"
Oct 29 21:57:51.274652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.274449    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7f7a1cf-25dlm\" (UID: \"b091f9da-2dd9-466b-b7b1-31b685b71481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"
Oct 29 21:57:51.375942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.375843    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47hjh\" (UniqueName: \"kubernetes.io/projected/b091f9da-2dd9-466b-b7b1-31b685b71481-kube-api-access-47hjh\") pod \"maas-api-revoke-sub-b7f7a1cf-25dlm\" (UID: \"b091f9da-2dd9-466b-b7b1-31b685b71481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"
Oct 29 21:57:51.375942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.375915    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7f7a1cf-25dlm\" (UID: \"b091f9da-2dd9-466b-b7b1-31b685b71481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"
Oct 29 21:57:51.376149 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.376068    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7f7a1cf-25dlm\" (UID: \"b091f9da-2dd9-466b-b7b1-31b685b71481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"
Oct 29 21:57:51.376660 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.376619    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7f7a1cf-25dlm\" (UID: \"b091f9da-2dd9-466b-b7b1-31b685b71481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"
Oct 29 21:57:51.378521 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.378503    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7f7a1cf-25dlm\" (UID: \"b091f9da-2dd9-466b-b7b1-31b685b71481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"
Oct 29 21:57:51.416574 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.416510    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47hjh\" (UniqueName: \"kubernetes.io/projected/b091f9da-2dd9-466b-b7b1-31b685b71481-kube-api-access-47hjh\") pod \"maas-api-revoke-sub-b7f7a1cf-25dlm\" (UID: \"b091f9da-2dd9-466b-b7b1-31b685b71481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"
Oct 29 21:57:51.549592 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.549560    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"
Oct 29 21:57:51.897938 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:51.897848    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"]
Oct 29 21:57:52.070515 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:52.070490    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:57:52.070617 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:52.070539    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 21:57:52.157418 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:52.157323    7082 generic.go:356] "Generic (PLEG): container finished" podID="ed5bbe37-aae9-4c32-96a0-dbc09a566fd6" containerID="85cbffe183e21c21296dba518043aea9c36fc408cf5f49b20dba4fcbf779c785" exitCode=0
Oct 29 21:57:52.157872 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:52.157411    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds" event={"ID":"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6","Type":"ContainerDied","Data":"85cbffe183e21c21296dba518043aea9c36fc408cf5f49b20dba4fcbf779c785"}
Oct 29 21:57:52.159000 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:52.158977    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm" event={"ID":"b091f9da-2dd9-466b-b7b1-31b685b71481","Type":"ContainerStarted","Data":"93e7880732b9bb0f93b52242339b9d8acf60c3bdb8b994d89ca6f2f6bfe1b56a"}
Oct 29 21:57:53.069682 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.069650    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:57:53.069852 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.069702    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 21:57:53.166938 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.166902    7082 generic.go:356] "Generic (PLEG): container finished" podID="b091f9da-2dd9-466b-b7b1-31b685b71481" containerID="0dea48477c192d880dcbde964bec610d3db6af09ad7ecebd4fdd64ba7bdde27c" exitCode=0
Oct 29 21:57:53.167263 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.166939    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm" event={"ID":"b091f9da-2dd9-466b-b7b1-31b685b71481","Type":"ContainerDied","Data":"0dea48477c192d880dcbde964bec610d3db6af09ad7ecebd4fdd64ba7bdde27c"}
Oct 29 21:57:53.378240 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.378215    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"
Oct 29 21:57:53.538683 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.538643    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-kube-api-access-5nj29\" (UniqueName: \"kubernetes.io/projected/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-kube-api-access-5nj29\") pod \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\" (UID: \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\") "
Oct 29 21:57:53.539024 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.538997    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-openshift-ca\") pod \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\" (UID: \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\") "
Oct 29 21:57:53.539206 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.539186    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-xks-ca\") pod \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\" (UID: \"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6\") "
Oct 29 21:57:53.541184 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.541157    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-openshift-ca" pod "ed5bbe37-aae9-4c32-96a0-dbc09a566fd6" (UID: "ed5bbe37-aae9-4c32-96a0-dbc09a566fd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:57:53.544616 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.544577    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-kube-api-access-5nj29" pod "ed5bbe37-aae9-4c32-96a0-dbc09a566fd6" (UID: "ed5bbe37-aae9-4c32-96a0-dbc09a566fd6"). InnerVolumeSpecName "kube-api-access-5nj29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:57:53.554046 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.554011    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nj29\" (UniqueName: \"kubernetes.io/projected/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-kube-api-access-5nj29\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:53.554046 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.554048    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:53.555308 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.555129    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-xks-ca" pod "ed5bbe37-aae9-4c32-96a0-dbc09a566fd6" (UID: "ed5bbe37-aae9-4c32-96a0-dbc09a566fd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:57:53.655345 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:53.655310    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:54.069913 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.069879    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:57:54.070082 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.069935    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 21:57:54.177277 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.177238    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds" event={"ID":"ed5bbe37-aae9-4c32-96a0-dbc09a566fd6","Type":"ContainerDied","Data":"2f6f65f576da998660e38e75f26bed4df4dc8fda9849a3685d577681af8e38e4"}
Oct 29 21:57:54.177683 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.177293    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f6f65f576da998660e38e75f26bed4df4dc8fda9849a3685d577681af8e38e4"
Oct 29 21:57:54.177683 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.177369    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"
Oct 29 21:57:54.373432 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.373408    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"
Oct 29 21:57:54.567213 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.567181    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-openshift-ca\") pod \"b091f9da-2dd9-466b-b7b1-31b685b71481\" (UID: \"b091f9da-2dd9-466b-b7b1-31b685b71481\") "
Oct 29 21:57:54.567416 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.567273    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b091f9da-2dd9-466b-b7b1-31b685b71481-kube-api-access-47hjh\" (UniqueName: \"kubernetes.io/projected/b091f9da-2dd9-466b-b7b1-31b685b71481-kube-api-access-47hjh\") pod \"b091f9da-2dd9-466b-b7b1-31b685b71481\" (UID: \"b091f9da-2dd9-466b-b7b1-31b685b71481\") "
Oct 29 21:57:54.567416 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.567336    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-xks-ca\") pod \"b091f9da-2dd9-466b-b7b1-31b685b71481\" (UID: \"b091f9da-2dd9-466b-b7b1-31b685b71481\") "
Oct 29 21:57:54.567651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.567599    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-openshift-ca" pod "b091f9da-2dd9-466b-b7b1-31b685b71481" (UID: "b091f9da-2dd9-466b-b7b1-31b685b71481"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:57:54.567815 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.567798    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:54.569450 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.569427    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b091f9da-2dd9-466b-b7b1-31b685b71481-kube-api-access-47hjh" pod "b091f9da-2dd9-466b-b7b1-31b685b71481" (UID: "b091f9da-2dd9-466b-b7b1-31b685b71481"). InnerVolumeSpecName "kube-api-access-47hjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:57:54.569541 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.569523    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-xks-ca" pod "b091f9da-2dd9-466b-b7b1-31b685b71481" (UID: "b091f9da-2dd9-466b-b7b1-31b685b71481"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:57:54.669550 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.669471    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47hjh\" (UniqueName: \"kubernetes.io/projected/b091f9da-2dd9-466b-b7b1-31b685b71481-kube-api-access-47hjh\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:54.669550 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:54.669501    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b091f9da-2dd9-466b-b7b1-31b685b71481-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:57:55.069573 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:55.069542    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:57:55.069740 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:55.069602    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 21:57:55.220226 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:55.220175    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm" event={"ID":"b091f9da-2dd9-466b-b7b1-31b685b71481","Type":"ContainerDied","Data":"93e7880732b9bb0f93b52242339b9d8acf60c3bdb8b994d89ca6f2f6bfe1b56a"}
Oct 29 21:57:55.220776 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:55.220240    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93e7880732b9bb0f93b52242339b9d8acf60c3bdb8b994d89ca6f2f6bfe1b56a"
Oct 29 21:57:55.220776 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:55.220312    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"
Oct 29 21:57:56.070007 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:56.069965    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:57:56.070201 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:56.070040    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 29 21:57:57.074111 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:57.074078    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:57.074747 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:57.074531    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:57.075325 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:57:57.075305    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 21:57:58.412199 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:57:58.412161    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:58:01.926650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:01.926027    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:58:04.960649 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:04.960427    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"]
Oct 29 21:58:04.967650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:04.961364    7082 cpu_manager.go:414] "containerMap: removing container" podUID="b091f9da-2dd9-466b-b7b1-31b685b71481" containerName="revoke-keys"
Oct 29 21:58:04.967650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:04.961390    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b091f9da-2dd9-466b-b7b1-31b685b71481" containerName="revoke-keys"
Oct 29 21:58:04.967650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:04.961472    7082 cpu_manager.go:414] "containerMap: removing container" podUID="ed5bbe37-aae9-4c32-96a0-dbc09a566fd6" containerName="revoke-keys"
Oct 29 21:58:04.967650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:04.961481    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5bbe37-aae9-4c32-96a0-dbc09a566fd6" containerName="revoke-keys"
Oct 29 21:58:04.967650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:04.961621    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5bbe37-aae9-4c32-96a0-dbc09a566fd6" containerName="revoke-keys"
Oct 29 21:58:04.967650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:04.961657    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="b091f9da-2dd9-466b-b7b1-31b685b71481" containerName="revoke-keys"
Oct 29 21:58:04.988651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:04.983893    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.053165 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.053129    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"]
Oct 29 21:58:05.055518 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.055403    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-743a5e-openshift-default-dockercfg-rnvpp\""
Oct 29 21:58:05.171269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.170416    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.171269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.170482    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1ecb02-ee43-4619-9333-a16d536102f8-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.171269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.170528    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.171269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.170670    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.171269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.170710    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckpmt\" (UniqueName: \"kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-kube-api-access-ckpmt\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.171269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.170804    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.171269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.170877    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.171269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.170989    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.171269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.171056    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.271913 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.271813    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.271913 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.271880    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.272168 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.271927    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.272168 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.271967    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.272168 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.271996    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1ecb02-ee43-4619-9333-a16d536102f8-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.272168 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.272033    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.272168 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.272125    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.272168 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.272165    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckpmt\" (UniqueName: \"kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-kube-api-access-ckpmt\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.273034 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.272240    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.273034 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.272733    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.273034 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.273008    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.273542 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.273247    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.273542 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.273488    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.275565 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.274711    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1ecb02-ee43-4619-9333-a16d536102f8-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.292402 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.292370    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.297160 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.296884    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.332915 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.332882    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.333812 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.333784    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckpmt\" (UniqueName: \"kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-kube-api-access-ckpmt\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.383535 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.383491    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:05.859651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.859589    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"]
Oct 29 21:58:05.860124 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:58:05.860099    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1ecb02_ee43_4619_9333_a16d536102f8.slice/crio-745405b54e2c0f2bde6687ab014320252bfb0d1e70abe7d0e6fec1186cec1063 WatchSource:0}: Error finding container 745405b54e2c0f2bde6687ab014320252bfb0d1e70abe7d0e6fec1186cec1063: Status 404 returned error can't find the container with id 745405b54e2c0f2bde6687ab014320252bfb0d1e70abe7d0e6fec1186cec1063
Oct 29 21:58:05.862877 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.862842    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:58:05.862985 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.862906    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:58:05.862985 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:05.862935    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:58:06.390706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:06.390663    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" event={"ID":"1f1ecb02-ee43-4619-9333-a16d536102f8","Type":"ContainerStarted","Data":"d1676651cfffbb392920180808eb75a168a8783429a5c2e286d541bbdaf6e009"}
Oct 29 21:58:06.391207 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:06.390749    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" event={"ID":"1f1ecb02-ee43-4619-9333-a16d536102f8","Type":"ContainerStarted","Data":"745405b54e2c0f2bde6687ab014320252bfb0d1e70abe7d0e6fec1186cec1063"}
Oct 29 21:58:06.481475 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:06.481402    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" podStartSLOduration=2.481373508 podStartE2EDuration="2.481373508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:06.451129824 +0000 UTC m=+658.951172125" watchObservedRunningTime="2026-10-29 21:58:06.481373508 +0000 UTC m=+658.981415810"
Oct 29 21:58:07.384063 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.384021    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:07.385766 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.385733    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 29 21:58:07.386017 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.385989    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 29 21:58:07.617464 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.617045    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"]
Oct 29 21:58:07.622404 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.622383    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.641452 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.641423    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"]
Oct 29 21:58:07.727119 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.727059    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.727340 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.727160    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.727340 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.727202    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.727340 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.727253    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.727340 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.727276    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.727340 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.727298    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3969f40e-b7c2-4839-85ef-998ab9d08122-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.727670 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.727345    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq7z2\" (UniqueName: \"kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-kube-api-access-sq7z2\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.727670 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.727414    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.727670 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.727567    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.828716 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.828676    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.828922 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.828751    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.828922 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.828798    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.828922 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.828832    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.828922 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.828880    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.828922 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.828901    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.829201 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.828921    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3969f40e-b7c2-4839-85ef-998ab9d08122-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.829201 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.828970    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq7z2\" (UniqueName: \"kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-kube-api-access-sq7z2\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.829201 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.829039    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.829790 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.829766    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.829955 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.829840    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.830950 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.830926    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3969f40e-b7c2-4839-85ef-998ab9d08122-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.831201 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.831179    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.831328 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.831305    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.832795 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.832772    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.833512 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.833482    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.852488 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.852463    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.852659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.852529    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq7z2\" (UniqueName: \"kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-kube-api-access-sq7z2\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:07.949964 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:07.949877    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:08.333585 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:08.333544    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"]
Oct 29 21:58:08.341218 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:58:08.341183    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3969f40e_b7c2_4839_85ef_998ab9d08122.slice/crio-88d5403152836dc7510c74273a98bea5b55a66ca2e1d461b50cc73cf67f42b9c WatchSource:0}: Error finding container 88d5403152836dc7510c74273a98bea5b55a66ca2e1d461b50cc73cf67f42b9c: Status 404 returned error can't find the container with id 88d5403152836dc7510c74273a98bea5b55a66ca2e1d461b50cc73cf67f42b9c
Oct 29 21:58:08.344607 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:08.344565    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:58:08.344709 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:08.344690    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:58:08.344769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:08.344740    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:58:08.403652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:08.384615    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 29 21:58:08.403652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:08.389969    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 29 21:58:08.448033 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:08.447911    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" event={"ID":"3969f40e-b7c2-4839-85ef-998ab9d08122","Type":"ContainerStarted","Data":"88d5403152836dc7510c74273a98bea5b55a66ca2e1d461b50cc73cf67f42b9c"}
Oct 29 21:58:09.384266 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:09.384233    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 29 21:58:09.384715 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:09.384286    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 29 21:58:09.458307 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:09.458274    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" event={"ID":"3969f40e-b7c2-4839-85ef-998ab9d08122","Type":"ContainerStarted","Data":"8ce199d2fcbefb880766e3756239e96330e08912f3b858f9ab8d7a9b8d706065"}
Oct 29 21:58:09.515653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:09.514788    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" podStartSLOduration=2.514766598 podStartE2EDuration="2.514766598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:09.509502306 +0000 UTC m=+662.009544611" watchObservedRunningTime="2026-10-29 21:58:09.514766598 +0000 UTC m=+662.014808899"
Oct 29 21:58:09.961654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:09.957274    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 21:58:09.961654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:09.957350    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 21:58:09.961654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:09.957672    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:10.384203 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:10.384171    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 29 21:58:10.384378 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:10.384225    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 29 21:58:10.952664 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:10.950366    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 21:58:10.952664 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:10.950443    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 21:58:11.384737 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:11.384701    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 29 21:58:11.385330 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:11.385294    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 29 21:58:11.950861 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:11.950820    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 21:58:11.951049 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:11.950879    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 21:58:12.384192 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:12.384152    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 29 21:58:12.384377 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:12.384215    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 29 21:58:12.950622 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:12.950586    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 21:58:12.951016 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:12.950683    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 21:58:13.461098 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:13.456938    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:13.461098 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:13.458152    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:13.461098 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:13.461048    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 21:58:13.951160 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:13.951124    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 21:58:13.951529 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:13.951195    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 21:58:14.951141 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:14.950807    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Oct 29 21:58:14.951141 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:14.950881    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Oct 29 21:58:15.955335 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:15.955306    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:15.955790 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:15.955768    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:15.956501 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:15.956483    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 21:58:28.412590 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:58:28.412552    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:58:29.597990 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:29.597950    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:58:33.884654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:33.884286    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 21:58:34.218825 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:34.218719    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 21:58:34.407798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:34.407751    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 21:58:34.408121 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:34.408086    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:58:38.482070 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.482027    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"]
Oct 29 21:58:38.489152 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.489125    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.516296 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.516265    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a940bc\"/\"distinct-w6-a940bc-kserve-self-signed-certs\""
Oct 29 21:58:38.521811 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.521780    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"]
Oct 29 21:58:38.553545 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.553510    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-model-cache\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.553722 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.553675    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qv85\" (UniqueName: \"kubernetes.io/projected/09277a7d-b7f4-4584-b6d6-8919612bcc77-kube-api-access-8qv85\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.553768 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.553733    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09277a7d-b7f4-4584-b6d6-8919612bcc77-tls-certs\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.553853 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.553839    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-home\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.553923 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.553896    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-dshm\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.553993 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.553964    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-tmp-dir\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.664781 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.664734    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-model-cache\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.669123 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.669078    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qv85\" (UniqueName: \"kubernetes.io/projected/09277a7d-b7f4-4584-b6d6-8919612bcc77-kube-api-access-8qv85\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.669459 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.669428    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09277a7d-b7f4-4584-b6d6-8919612bcc77-tls-certs\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.669777 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.669754    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-home\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.670095 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.670070    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-dshm\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.670237 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.670220    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-tmp-dir\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.670976 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.670955    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-tmp-dir\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.671954 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.666986    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-model-cache\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.672102 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.671545    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-home\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.675022 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.674997    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-dshm\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.686280 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.686252    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09277a7d-b7f4-4584-b6d6-8919612bcc77-tls-certs\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.709684 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.709649    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qv85\" (UniqueName: \"kubernetes.io/projected/09277a7d-b7f4-4584-b6d6-8919612bcc77-kube-api-access-8qv85\") pod \"distinct-w6-a940bc-kserve-55cc665cb6-mm9xl\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") " pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:38.805715 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:38.805680    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:39.122819 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:39.122790    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"]
Oct 29 21:58:39.123995 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:58:39.123965    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09277a7d_b7f4_4584_b6d6_8919612bcc77.slice/crio-11ad5fb97818fd03fe8f20f9e3d1a2d749a422c3a5fed62728ccfd1117ef8063 WatchSource:0}: Error finding container 11ad5fb97818fd03fe8f20f9e3d1a2d749a422c3a5fed62728ccfd1117ef8063: Status 404 returned error can't find the container with id 11ad5fb97818fd03fe8f20f9e3d1a2d749a422c3a5fed62728ccfd1117ef8063
Oct 29 21:58:39.933272 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:39.933232    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" event={"ID":"09277a7d-b7f4-4584-b6d6-8919612bcc77","Type":"ContainerStarted","Data":"5bce4c30aacd93665628a05069fd9615c4e7a8a9239ae627620a80cd35a2b004"}
Oct 29 21:58:39.933755 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:39.933284    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" event={"ID":"09277a7d-b7f4-4584-b6d6-8919612bcc77","Type":"ContainerStarted","Data":"11ad5fb97818fd03fe8f20f9e3d1a2d749a422c3a5fed62728ccfd1117ef8063"}
Oct 29 21:58:39.978418 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:39.978370    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podStartSLOduration=1.978354071 podStartE2EDuration="1.978354071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:39.976118627 +0000 UTC m=+692.476160927" watchObservedRunningTime="2026-10-29 21:58:39.978354071 +0000 UTC m=+692.478396370"
Oct 29 21:58:40.806788 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:40.806754    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:46.333893 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.333858    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"]
Oct 29 21:58:46.339354 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.339337    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"
Oct 29 21:58:46.356471 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.356444    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"]
Oct 29 21:58:46.364094 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.364070    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 21:58:46.364308 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.364124    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 21:58:46.364308 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.364146    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 21:58:46.364308 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.364205    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 21:58:46.399266 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.399236    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44867098-f8zhg\" (UID: \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"
Oct 29 21:58:46.399402 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.399307    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44867098-f8zhg\" (UID: \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"
Oct 29 21:58:46.399462 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.399408    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwcpd\" (UniqueName: \"kubernetes.io/projected/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-kube-api-access-pwcpd\") pod \"maas-api-revoke-sub-44867098-f8zhg\" (UID: \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"
Oct 29 21:58:46.405060 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.405034    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"]
Oct 29 21:58:46.409199 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.409184    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"
Oct 29 21:58:46.430236 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.430217    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"]
Oct 29 21:58:46.500003 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.499968    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b337d2-wg7lb\" (UID: \"2a110df2-476c-4d68-ab62-5605e00cdeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"
Oct 29 21:58:46.500167 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.500020    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcmlr\" (UniqueName: \"kubernetes.io/projected/2a110df2-476c-4d68-ab62-5605e00cdeee-kube-api-access-zcmlr\") pod \"maas-api-revoke-sub-d9b337d2-wg7lb\" (UID: \"2a110df2-476c-4d68-ab62-5605e00cdeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"
Oct 29 21:58:46.500216 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.500158    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44867098-f8zhg\" (UID: \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"
Oct 29 21:58:46.500269 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.500245    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44867098-f8zhg\" (UID: \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"
Oct 29 21:58:46.500357 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.500340    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwcpd\" (UniqueName: \"kubernetes.io/projected/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-kube-api-access-pwcpd\") pod \"maas-api-revoke-sub-44867098-f8zhg\" (UID: \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"
Oct 29 21:58:46.500413 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.500375    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b337d2-wg7lb\" (UID: \"2a110df2-476c-4d68-ab62-5605e00cdeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"
Oct 29 21:58:46.500947 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.500925    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44867098-f8zhg\" (UID: \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"
Oct 29 21:58:46.502711 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.502690    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44867098-f8zhg\" (UID: \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"
Oct 29 21:58:46.523918 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.523892    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwcpd\" (UniqueName: \"kubernetes.io/projected/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-kube-api-access-pwcpd\") pod \"maas-api-revoke-sub-44867098-f8zhg\" (UID: \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"
Oct 29 21:58:46.601225 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.601146    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b337d2-wg7lb\" (UID: \"2a110df2-476c-4d68-ab62-5605e00cdeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"
Oct 29 21:58:46.601225 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.601214    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcmlr\" (UniqueName: \"kubernetes.io/projected/2a110df2-476c-4d68-ab62-5605e00cdeee-kube-api-access-zcmlr\") pod \"maas-api-revoke-sub-d9b337d2-wg7lb\" (UID: \"2a110df2-476c-4d68-ab62-5605e00cdeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"
Oct 29 21:58:46.601413 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.601393    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b337d2-wg7lb\" (UID: \"2a110df2-476c-4d68-ab62-5605e00cdeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"
Oct 29 21:58:46.602007 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.601985    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b337d2-wg7lb\" (UID: \"2a110df2-476c-4d68-ab62-5605e00cdeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"
Oct 29 21:58:46.603563 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.603534    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b337d2-wg7lb\" (UID: \"2a110df2-476c-4d68-ab62-5605e00cdeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"
Oct 29 21:58:46.637842 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.637821    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcmlr\" (UniqueName: \"kubernetes.io/projected/2a110df2-476c-4d68-ab62-5605e00cdeee-kube-api-access-zcmlr\") pod \"maas-api-revoke-sub-d9b337d2-wg7lb\" (UID: \"2a110df2-476c-4d68-ab62-5605e00cdeee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"
Oct 29 21:58:46.653849 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.653828    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"
Oct 29 21:58:46.724135 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.722971    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"
Oct 29 21:58:46.965596 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.965565    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"]
Oct 29 21:58:46.967468 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:58:46.967323    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360c6c19_fe30_47d9_8bc6_3e7fe73859c1.slice/crio-a34713d9ff93e15cf6a98b0717ffc08ed7ad8b99b056c6d20c323428a86c961f WatchSource:0}: Error finding container a34713d9ff93e15cf6a98b0717ffc08ed7ad8b99b056c6d20c323428a86c961f: Status 404 returned error can't find the container with id a34713d9ff93e15cf6a98b0717ffc08ed7ad8b99b056c6d20c323428a86c961f
Oct 29 21:58:46.993923 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:46.993895    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg" event={"ID":"360c6c19-fe30-47d9-8bc6-3e7fe73859c1","Type":"ContainerStarted","Data":"a34713d9ff93e15cf6a98b0717ffc08ed7ad8b99b056c6d20c323428a86c961f"}
Oct 29 21:58:47.062982 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:47.045260    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"]
Oct 29 21:58:47.070887 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:58:47.070838    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a110df2_476c_4d68_ab62_5605e00cdeee.slice/crio-a000a1352c710490c9a93ca4c6c55c379d7558f25543d87dcb580c67ba68b5dc WatchSource:0}: Error finding container a000a1352c710490c9a93ca4c6c55c379d7558f25543d87dcb580c67ba68b5dc: Status 404 returned error can't find the container with id a000a1352c710490c9a93ca4c6c55c379d7558f25543d87dcb580c67ba68b5dc
Oct 29 21:58:48.005417 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:48.005388    7082 generic.go:356] "Generic (PLEG): container finished" podID="360c6c19-fe30-47d9-8bc6-3e7fe73859c1" containerID="57691b086424e890e296c68505e5cd090b3027d5a8b3b66fcbb2643dc919174b" exitCode=0
Oct 29 21:58:48.005813 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:48.005470    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg" event={"ID":"360c6c19-fe30-47d9-8bc6-3e7fe73859c1","Type":"ContainerDied","Data":"57691b086424e890e296c68505e5cd090b3027d5a8b3b66fcbb2643dc919174b"}
Oct 29 21:58:48.007189 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:48.007168    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb" event={"ID":"2a110df2-476c-4d68-ab62-5605e00cdeee","Type":"ContainerStarted","Data":"295a2746ba7f6a52c281a38ba71b26758e4231249eaa99be26bfc1c720dd6389"}
Oct 29 21:58:48.007298 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:48.007212    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb" event={"ID":"2a110df2-476c-4d68-ab62-5605e00cdeee","Type":"ContainerStarted","Data":"a000a1352c710490c9a93ca4c6c55c379d7558f25543d87dcb580c67ba68b5dc"}
Oct 29 21:58:48.089616 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:48.089561    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb" podStartSLOduration=2.089543328 podStartE2EDuration="2.089543328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:48.088750148 +0000 UTC m=+700.588792472" watchObservedRunningTime="2026-10-29 21:58:48.089543328 +0000 UTC m=+700.589585630"
Oct 29 21:58:48.806920 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:48.806895    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:48.819604 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:48.819579    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:48.957376 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:48.957343    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"]
Oct 29 21:58:48.965599 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:48.965564    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:48.993473 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:48.993445    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"]
Oct 29 21:58:49.023459 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.023390    7082 generic.go:356] "Generic (PLEG): container finished" podID="2a110df2-476c-4d68-ab62-5605e00cdeee" containerID="295a2746ba7f6a52c281a38ba71b26758e4231249eaa99be26bfc1c720dd6389" exitCode=0
Oct 29 21:58:49.025260 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.025227    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb" event={"ID":"2a110df2-476c-4d68-ab62-5605e00cdeee","Type":"ContainerDied","Data":"295a2746ba7f6a52c281a38ba71b26758e4231249eaa99be26bfc1c720dd6389"}
Oct 29 21:58:49.034374 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.034342    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.034503 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.034414    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.034503 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.034467    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.034615 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.034514    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbw57\" (UniqueName: \"kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-kube-api-access-rbw57\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.034615 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.034559    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.034833 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.034641    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.034833 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.034755    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.034833 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.034779    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca6221e-5b57-49a7-8a18-799752edb517-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.034984 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.034847    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca6221e-5b57-49a7-8a18-799752edb517-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.040519 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.040496    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:58:49.135600 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.135567    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.135822 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.135655    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbw57\" (UniqueName: \"kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-kube-api-access-rbw57\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.135822 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.135689    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.135822 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.135753    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.136009 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.135844    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.136009 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.135870    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca6221e-5b57-49a7-8a18-799752edb517-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.136009 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.135918    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca6221e-5b57-49a7-8a18-799752edb517-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.136009 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.135993    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.136244 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.136046    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.136875 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.136477    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.136875 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.136773    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.136875 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.136822    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.137141 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.136900    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca6221e-5b57-49a7-8a18-799752edb517-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.137141 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.136948    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.139826 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.139802    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca6221e-5b57-49a7-8a18-799752edb517-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.153295 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.153266    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.162524 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.162495    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.163935 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.163898    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbw57\" (UniqueName: \"kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-kube-api-access-rbw57\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.200430 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.200408    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"
Oct 29 21:58:49.236705 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.236665    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-openshift-ca\") pod \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\" (UID: \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\") "
Oct 29 21:58:49.236889 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.236834    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-kube-api-access-pwcpd\" (UniqueName: \"kubernetes.io/projected/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-kube-api-access-pwcpd\") pod \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\" (UID: \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\") "
Oct 29 21:58:49.236959 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.236893    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-xks-ca\") pod \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\" (UID: \"360c6c19-fe30-47d9-8bc6-3e7fe73859c1\") "
Oct 29 21:58:49.237029 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.237006    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-openshift-ca" pod "360c6c19-fe30-47d9-8bc6-3e7fe73859c1" (UID: "360c6c19-fe30-47d9-8bc6-3e7fe73859c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:58:49.237369 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.237351    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:58:49.239022 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.239000    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-kube-api-access-pwcpd" pod "360c6c19-fe30-47d9-8bc6-3e7fe73859c1" (UID: "360c6c19-fe30-47d9-8bc6-3e7fe73859c1"). InnerVolumeSpecName "kube-api-access-pwcpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:58:49.239077 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.239046    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-xks-ca" pod "360c6c19-fe30-47d9-8bc6-3e7fe73859c1" (UID: "360c6c19-fe30-47d9-8bc6-3e7fe73859c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:58:49.285874 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.285660    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:49.338460 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.338430    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwcpd\" (UniqueName: \"kubernetes.io/projected/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-kube-api-access-pwcpd\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:58:49.338460 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.338462    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/360c6c19-fe30-47d9-8bc6-3e7fe73859c1-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:58:49.634030 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.634007    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"]
Oct 29 21:58:49.635245 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:58:49.635217    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca6221e_5b57_49a7_8a18_799752edb517.slice/crio-34df19976a1970d58fba485cf77adf751824c62e6864d7eb11b005403715156d WatchSource:0}: Error finding container 34df19976a1970d58fba485cf77adf751824c62e6864d7eb11b005403715156d: Status 404 returned error can't find the container with id 34df19976a1970d58fba485cf77adf751824c62e6864d7eb11b005403715156d
Oct 29 21:58:49.638019 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.637977    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:58:49.638088 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.638068    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:58:49.638132 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:49.638116    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:58:50.032702 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.032597    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" event={"ID":"fca6221e-5b57-49a7-8a18-799752edb517","Type":"ContainerStarted","Data":"9e600ad115f340536d61e56ed2d7473645968cf243a1a9a3e895485c4874b8bf"}
Oct 29 21:58:50.032702 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.032693    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" event={"ID":"fca6221e-5b57-49a7-8a18-799752edb517","Type":"ContainerStarted","Data":"34df19976a1970d58fba485cf77adf751824c62e6864d7eb11b005403715156d"}
Oct 29 21:58:50.034303 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.034274    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"
Oct 29 21:58:50.034423 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.034334    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg" event={"ID":"360c6c19-fe30-47d9-8bc6-3e7fe73859c1","Type":"ContainerDied","Data":"a34713d9ff93e15cf6a98b0717ffc08ed7ad8b99b056c6d20c323428a86c961f"}
Oct 29 21:58:50.034423 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.034372    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a34713d9ff93e15cf6a98b0717ffc08ed7ad8b99b056c6d20c323428a86c961f"
Oct 29 21:58:50.081325 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.081265    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podStartSLOduration=2.08124255 podStartE2EDuration="2.08124255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:50.078435257 +0000 UTC m=+702.578477557" watchObservedRunningTime="2026-10-29 21:58:50.08124255 +0000 UTC m=+702.581284851"
Oct 29 21:58:50.199476 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.199446    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"
Oct 29 21:58:50.251723 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.249536    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-xks-ca\") pod \"2a110df2-476c-4d68-ab62-5605e00cdeee\" (UID: \"2a110df2-476c-4d68-ab62-5605e00cdeee\") "
Oct 29 21:58:50.251723 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.249574    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a110df2-476c-4d68-ab62-5605e00cdeee-kube-api-access-zcmlr\" (UniqueName: \"kubernetes.io/projected/2a110df2-476c-4d68-ab62-5605e00cdeee-kube-api-access-zcmlr\") pod \"2a110df2-476c-4d68-ab62-5605e00cdeee\" (UID: \"2a110df2-476c-4d68-ab62-5605e00cdeee\") "
Oct 29 21:58:50.251723 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.249690    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-openshift-ca\") pod \"2a110df2-476c-4d68-ab62-5605e00cdeee\" (UID: \"2a110df2-476c-4d68-ab62-5605e00cdeee\") "
Oct 29 21:58:50.251723 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.250035    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-openshift-ca" pod "2a110df2-476c-4d68-ab62-5605e00cdeee" (UID: "2a110df2-476c-4d68-ab62-5605e00cdeee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:58:50.251723 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.250233    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:58:50.258270 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.258177    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-xks-ca" pod "2a110df2-476c-4d68-ab62-5605e00cdeee" (UID: "2a110df2-476c-4d68-ab62-5605e00cdeee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:58:50.258413 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.258316    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a110df2-476c-4d68-ab62-5605e00cdeee-kube-api-access-zcmlr" pod "2a110df2-476c-4d68-ab62-5605e00cdeee" (UID: "2a110df2-476c-4d68-ab62-5605e00cdeee"). InnerVolumeSpecName "kube-api-access-zcmlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:58:50.305714 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.289839    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:50.316974 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.316773    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 21:58:50.317169 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.316863    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 21:58:50.351320 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.351291    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a110df2-476c-4d68-ab62-5605e00cdeee-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:58:50.351320 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:50.351317    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcmlr\" (UniqueName: \"kubernetes.io/projected/2a110df2-476c-4d68-ab62-5605e00cdeee-kube-api-access-zcmlr\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 21:58:51.043572 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:51.043535    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb" event={"ID":"2a110df2-476c-4d68-ab62-5605e00cdeee","Type":"ContainerDied","Data":"a000a1352c710490c9a93ca4c6c55c379d7558f25543d87dcb580c67ba68b5dc"}
Oct 29 21:58:51.043572 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:51.043567    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"
Oct 29 21:58:51.044059 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:51.043592    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a000a1352c710490c9a93ca4c6c55c379d7558f25543d87dcb580c67ba68b5dc"
Oct 29 21:58:51.314598 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:51.314549    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 21:58:51.314838 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:51.314650    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 21:58:52.288984 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:52.288946    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 21:58:52.289570 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:52.289538    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 21:58:53.294800 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:53.290284    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 21:58:53.294800 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:53.290340    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 21:58:54.293651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:54.293588    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 21:58:54.293936 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:54.293903    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 21:58:55.298654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:55.295728    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 21:58:55.298654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:55.295789    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 21:58:56.287012 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:56.286974    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 29 21:58:56.287239 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:56.287042    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 29 21:58:56.939889 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:56.939855    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:58:57.296709 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:57.295892    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:57.305692 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:57.297280    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:57.305692 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:58:57.298287    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 21:58:58.589005 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:58:58.588919    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:59:00.104659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.093307    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"]
Oct 29 21:59:00.104659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.094244    7082 cpu_manager.go:414] "containerMap: removing container" podUID="2a110df2-476c-4d68-ab62-5605e00cdeee" containerName="revoke-keys"
Oct 29 21:59:00.104659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.094266    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a110df2-476c-4d68-ab62-5605e00cdeee" containerName="revoke-keys"
Oct 29 21:59:00.104659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.094318    7082 cpu_manager.go:414] "containerMap: removing container" podUID="360c6c19-fe30-47d9-8bc6-3e7fe73859c1" containerName="revoke-keys"
Oct 29 21:59:00.104659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.094327    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360c6c19-fe30-47d9-8bc6-3e7fe73859c1" containerName="revoke-keys"
Oct 29 21:59:00.104659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.094496    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a110df2-476c-4d68-ab62-5605e00cdeee" containerName="revoke-keys"
Oct 29 21:59:00.104659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.094513    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="360c6c19-fe30-47d9-8bc6-3e7fe73859c1" containerName="revoke-keys"
Oct 29 21:59:00.123657 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.120673    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.142770 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.142739    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"]
Oct 29 21:59:00.383960 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.383833    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:00.385256 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.385186    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 21:59:00.582282 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.582239    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.585980 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.585943    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.586292 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.586269    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.595775 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.595728    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.596070 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.596037    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.599641 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.599596    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c530e17d-eb72-4b77-b2e6-669d8225b75a-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.619652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.607158    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.759723 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.758712    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.759723 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.758784    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvlmt\" (UniqueName: \"kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-kube-api-access-wvlmt\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.869654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.863409    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.869654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.863501    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.869654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.863570    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.869654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.863657    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.869654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.863695    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.869654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.863733    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c530e17d-eb72-4b77-b2e6-669d8225b75a-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.869654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.863821    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.869654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.863852    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.869654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.863883    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvlmt\" (UniqueName: \"kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-kube-api-access-wvlmt\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.869654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.864722    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.918654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.913430    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c530e17d-eb72-4b77-b2e6-669d8225b75a-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.918654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.913700    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.918654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.913940    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:00.918654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:00.914478    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:01.171652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:01.163233    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:01.171652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:01.169223    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:01.172205 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:01.171817    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvlmt\" (UniqueName: \"kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-kube-api-access-wvlmt\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:01.172519 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:01.172463    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:01.373708 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:01.373256    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:01.422157 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:01.411417    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 21:59:01.422157 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:01.412386    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:59:01.422157 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:01.411363    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:01.422157 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:01.414003    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:59:01.585145 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:01.583034    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:59:01.982804 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:01.982760    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:02.079959 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.079924    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 21:59:02.080251 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.080213    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:02.082060 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.082033    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:02.193488 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.193409    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:59:02.234663 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.234095    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Oct 29 21:59:02.302430 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.302392    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"]
Oct 29 21:59:02.323485 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.323458    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"]
Oct 29 21:59:02.327880 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.323711    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.329858 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.329827    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"]
Oct 29 21:59:02.329997 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.329929    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.336647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.334410    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"]
Oct 29 21:59:02.440656 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.430330    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.440944 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.440914    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvzjw\" (UniqueName: \"kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-kube-api-access-zvzjw\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.441162 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.441141    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.444805 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.444777    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.445017 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.444999    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4150595-96b9-4f34-a2be-fdb99526b429-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.475965 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.475931    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.492659 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.476255    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.493822 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.493793    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc27893e-a205-4145-9752-1e6813c3b88d-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.494076 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.494056    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.494253 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.494237    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.494417 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.494398    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.494574 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.494555    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc27893e-a205-4145-9752-1e6813c3b88d-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.494776 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.494758    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4150595-96b9-4f34-a2be-fdb99526b429-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.494963 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.494945    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.495153 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.495136    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.500737 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.500711    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8rvb\" (UniqueName: \"kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-kube-api-access-d8rvb\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.501033 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.501005    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.545407 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.545380    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689233    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689286    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvzjw\" (UniqueName: \"kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-kube-api-access-zvzjw\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689329    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689378    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689408    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4150595-96b9-4f34-a2be-fdb99526b429-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689455    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689482    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689552    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc27893e-a205-4145-9752-1e6813c3b88d-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689594    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689654    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689691    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689725    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc27893e-a205-4145-9752-1e6813c3b88d-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689751    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4150595-96b9-4f34-a2be-fdb99526b429-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689802    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689842    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689865    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8rvb\" (UniqueName: \"kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-kube-api-access-d8rvb\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.689935    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.690002    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.692244    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.693706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.692839    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.701414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.700873    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.701414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.701113    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.701414 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.701343    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.703114 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.702724    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4150595-96b9-4f34-a2be-fdb99526b429-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.703463 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.703365    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc27893e-a205-4145-9752-1e6813c3b88d-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.704449 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.704368    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.704730 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.704638    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.714199 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.713427    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:02.730730 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.730691    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.783128 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.781712    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.785399 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.783544    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.785399 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.785338    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.799219 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.797107    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.799219 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.797496    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc27893e-a205-4145-9752-1e6813c3b88d-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.799219 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.797749    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4150595-96b9-4f34-a2be-fdb99526b429-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.799219 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.798077    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvzjw\" (UniqueName: \"kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-kube-api-access-zvzjw\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.801396 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.801374    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8rvb\" (UniqueName: \"kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-kube-api-access-d8rvb\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:02.899694 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.896341    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:02.954908 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:02.954875    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:03.131686 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:03.131653    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"]
Oct 29 21:59:03.133052 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:59:03.133020    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc530e17d_eb72_4b77_b2e6_669d8225b75a.slice/crio-641b82ca2ac59d23def76979c48ad47a10e06d62e34c742e41ac908b1a597a4f WatchSource:0}: Error finding container 641b82ca2ac59d23def76979c48ad47a10e06d62e34c742e41ac908b1a597a4f: Status 404 returned error can't find the container with id 641b82ca2ac59d23def76979c48ad47a10e06d62e34c742e41ac908b1a597a4f
Oct 29 21:59:03.135486 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:03.135450    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:03.135597 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:03.135529    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:03.135597 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:03.135560    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:03.258714 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:03.258690    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"]
Oct 29 21:59:03.260314 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:59:03.260288    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc27893e_a205_4145_9752_1e6813c3b88d.slice/crio-e77bfc4ead9334d6f123b6c994b5f22de777e85f3ace2e88025c2661f1fcf30b WatchSource:0}: Error finding container e77bfc4ead9334d6f123b6c994b5f22de777e85f3ace2e88025c2661f1fcf30b: Status 404 returned error can't find the container with id e77bfc4ead9334d6f123b6c994b5f22de777e85f3ace2e88025c2661f1fcf30b
Oct 29 21:59:03.262951 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:03.262916    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:03.263061 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:03.262998    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:03.263061 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:03.263041    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:03.301151 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:03.301120    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"]
Oct 29 21:59:03.306316 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:59:03.306275    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4150595_96b9_4f34_a2be_fdb99526b429.slice/crio-f7338b0c6ae90a1fc14965d29a129eb9ed308c906061eb797d7ff371a19c965f WatchSource:0}: Error finding container f7338b0c6ae90a1fc14965d29a129eb9ed308c906061eb797d7ff371a19c965f: Status 404 returned error can't find the container with id f7338b0c6ae90a1fc14965d29a129eb9ed308c906061eb797d7ff371a19c965f
Oct 29 21:59:03.310596 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:03.310537    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:03.310731 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:03.310667    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:03.310731 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:03.310719    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:04.108649 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.103459    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" event={"ID":"f4150595-96b9-4f34-a2be-fdb99526b429","Type":"ContainerStarted","Data":"24ec1b352aef15539c438520cbd7f48b4672f3a51332a9e8e1708e57d01fbddf"}
Oct 29 21:59:04.108649 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.103528    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" event={"ID":"f4150595-96b9-4f34-a2be-fdb99526b429","Type":"ContainerStarted","Data":"f7338b0c6ae90a1fc14965d29a129eb9ed308c906061eb797d7ff371a19c965f"}
Oct 29 21:59:04.111871 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.111042    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" event={"ID":"dc27893e-a205-4145-9752-1e6813c3b88d","Type":"ContainerStarted","Data":"da5771cb6b543f720e7ed14cd08e7bb7feefd16bb63e31e2ac800898bde88b0d"}
Oct 29 21:59:04.111871 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.111102    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" event={"ID":"dc27893e-a205-4145-9752-1e6813c3b88d","Type":"ContainerStarted","Data":"e77bfc4ead9334d6f123b6c994b5f22de777e85f3ace2e88025c2661f1fcf30b"}
Oct 29 21:59:04.114167 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.114141    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" event={"ID":"c530e17d-eb72-4b77-b2e6-669d8225b75a","Type":"ContainerStarted","Data":"2a3b4996408a46bec23a8d3928e7dfb372c2008bb905f64c527637a4d748279f"}
Oct 29 21:59:04.114331 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.114316    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" event={"ID":"c530e17d-eb72-4b77-b2e6-669d8225b75a","Type":"ContainerStarted","Data":"641b82ca2ac59d23def76979c48ad47a10e06d62e34c742e41ac908b1a597a4f"}
Oct 29 21:59:04.178825 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.178758    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" podStartSLOduration=3.178738614 podStartE2EDuration="3.178738614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:04.165761392 +0000 UTC m=+716.665803703" watchObservedRunningTime="2026-10-29 21:59:04.178738614 +0000 UTC m=+716.678780914"
Oct 29 21:59:04.245676 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.241871    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podStartSLOduration=3.241846878 podStartE2EDuration="3.241846878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:04.222528688 +0000 UTC m=+716.722570989" watchObservedRunningTime="2026-10-29 21:59:04.241846878 +0000 UTC m=+716.741889181"
Oct 29 21:59:04.311791 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.311729    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podStartSLOduration=4.311707306 podStartE2EDuration="4.311707306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:04.308384181 +0000 UTC m=+716.808426481" watchObservedRunningTime="2026-10-29 21:59:04.311707306 +0000 UTC m=+716.811749608"
Oct 29 21:59:04.380829 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.380727    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:04.382021 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.382003    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 21:59:04.382150 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.382132    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 21:59:04.913541 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.913511    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:04.913907 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.913827    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 21:59:04.914095 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.914065    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 21:59:04.957326 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.957273    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:04.957706 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.957683    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 21:59:04.957885 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:04.957851    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 21:59:05.382995 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:05.382958    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 21:59:05.383923 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:05.383871    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 21:59:05.907651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:05.907110    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 21:59:05.907651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:05.907188    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 21:59:05.956652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:05.955857    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 21:59:05.956652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:05.955934    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 21:59:06.375079 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:06.375038    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 21:59:06.375409 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:06.375377    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 21:59:06.907681 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:06.907102    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 21:59:06.907681 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:06.907180    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 21:59:06.957656 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:06.957339    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 21:59:06.957656 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:06.957423    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 21:59:07.375653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:07.375289    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 21:59:07.375653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:07.375337    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 21:59:07.910602 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:07.909093    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 21:59:07.910602 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:07.909153    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 21:59:07.957866 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:07.957028    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 21:59:07.957866 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:07.957100    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 21:59:08.431564 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:08.431530    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 21:59:08.431971 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:08.431927    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 21:59:08.918667 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:08.901695    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 21:59:08.918667 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:08.901760    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 21:59:08.972650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:08.966832    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 21:59:08.972650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:08.966891    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 21:59:08.972650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:08.967115    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"]
Oct 29 21:59:08.986040 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:08.985827    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"]
Oct 29 21:59:08.996997 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:08.986585    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.132653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.131856    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.132653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.131920    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l97jv\" (UniqueName: \"kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-kube-api-access-l97jv\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.132653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.132004    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.132653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.132082    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.132653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.132139    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.132653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.132194    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.132653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.132294    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.132653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.132363    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.132653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.132401    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.253653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.236987    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.253653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.237066    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.253653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.237107    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l97jv\" (UniqueName: \"kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-kube-api-access-l97jv\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.253653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.237178    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.253653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.237238    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.253653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.237287    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.253653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.237336    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.253653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.237421    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.253653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.237481    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.297781 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.280541    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.297781 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.280836    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.297781 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.281520    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.297781 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.284322    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.297781 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.284562    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.337653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.310298    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.337653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.330035    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.382651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.378744    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 21:59:09.382651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.378809    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 21:59:09.398652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.391123    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l97jv\" (UniqueName: \"kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-kube-api-access-l97jv\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.426710 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.421205    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.546656 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.545763    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:09.904183 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.903541    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 21:59:09.909649 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.903602    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 21:59:09.962678 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.957128    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 21:59:09.962678 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:09.957204    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 21:59:10.370533 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:10.369657    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"]
Oct 29 21:59:10.406208 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:10.388735    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 29 21:59:10.406208 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:10.388799    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 29 21:59:10.487679 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:10.484610    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:10.487679 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:10.484734    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:10.487679 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:10.484783    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:10.520942 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:10.520505    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" event={"ID":"65dfc02a-c81d-4366-a1a5-a5808d335aaf","Type":"ContainerStarted","Data":"60882761c6bc8cef65b178f934791ba1389e544f21080fa60996b6fdca4aa977"}
Oct 29 21:59:10.903725 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:10.902264    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 29 21:59:10.903725 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:10.902323    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 29 21:59:10.956665 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:10.956290    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 29 21:59:10.956665 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:10.956349    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 29 21:59:11.554081 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:11.554039    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" event={"ID":"65dfc02a-c81d-4366-a1a5-a5808d335aaf","Type":"ContainerStarted","Data":"bccbd6fa084f2a6ee5713128fc2175c95a417810cbcba8583c24b8cce74c358e"}
Oct 29 21:59:11.883700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:11.878924    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:11.883700 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:11.880389    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:11.885494 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:11.884089    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 21:59:11.950605 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:11.950537    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" podStartSLOduration=3.9505145600000002 podStartE2EDuration="3.95051456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:11.654913056 +0000 UTC m=+724.154955356" watchObservedRunningTime="2026-10-29 21:59:11.95051456 +0000 UTC m=+724.450556860"
Oct 29 21:59:12.077680 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:12.069932    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:12.077680 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:12.071358    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:12.077680 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:12.072535    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 21:59:12.480494 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:12.480448    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:12.481373 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:12.481356    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:12.482021 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:12.481999    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 21:59:12.546177 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:12.546150    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:12.547925 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:12.547900    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 29 21:59:12.548062 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:12.547948    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 29 21:59:13.547122 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.547089    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 29 21:59:13.547525 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.547142    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 29 21:59:13.666712 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.666674    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"]
Oct 29 21:59:13.671463 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.671428    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.682075 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.682042    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-tmp-dir\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.682230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.682161    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9900ad31-588e-4077-bef3-1ee57af99f84-tls-certs\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.682230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.682215    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-model-cache\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.683142 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.683116    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tfhv\" (UniqueName: \"kubernetes.io/projected/9900ad31-588e-4077-bef3-1ee57af99f84-kube-api-access-7tfhv\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.683319 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.683179    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-dshm\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.683319 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.683228    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-home\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.725825 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.725792    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a9def7\"/\"openshift-service-ca.crt\""
Oct 29 21:59:13.726028 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.725935    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a9def7\"/\"kube-root-ca.crt\""
Oct 29 21:59:13.726104 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.726013    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a9def7\"/\"default-dockercfg-26fhs\""
Oct 29 21:59:13.726163 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.726129    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a9def7\"/\"distinct-w2-a9def7-kserve-self-signed-certs\""
Oct 29 21:59:13.731822 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.731789    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"]
Oct 29 21:59:13.784345 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.784310    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-home\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.784508 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.784400    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-tmp-dir\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.784508 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.784444    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9900ad31-588e-4077-bef3-1ee57af99f84-tls-certs\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.784508 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.784487    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-model-cache\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.784609 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.784540    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tfhv\" (UniqueName: \"kubernetes.io/projected/9900ad31-588e-4077-bef3-1ee57af99f84-kube-api-access-7tfhv\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.784609 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.784591    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-dshm\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.789678 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.784783    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-home\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.790873 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.790812    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-model-cache\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.791472 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.791450    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-tmp-dir\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.815585 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.815559    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-dshm\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.817368 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.817332    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9900ad31-588e-4077-bef3-1ee57af99f84-tls-certs\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:13.818260 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:13.818235    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tfhv\" (UniqueName: \"kubernetes.io/projected/9900ad31-588e-4077-bef3-1ee57af99f84-kube-api-access-7tfhv\") pod \"distinct-w2-a9def7-kserve-77bcbd688f-jpb2z\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") " pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:14.034924 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:14.034888    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:14.421777 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:14.421742    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"]
Oct 29 21:59:14.546735 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:14.546701    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 29 21:59:14.546864 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:14.546754    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 29 21:59:14.577336 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:14.577296    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" event={"ID":"9900ad31-588e-4077-bef3-1ee57af99f84","Type":"ContainerStarted","Data":"70664f7b0631ef941b485d77c1a48c04119bd96fbee915ba3a22a387a5b7cabb"}
Oct 29 21:59:14.577884 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:14.577344    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" event={"ID":"9900ad31-588e-4077-bef3-1ee57af99f84","Type":"ContainerStarted","Data":"eaa2c17d461f24e4ee1557cd2a9acaf1fad8b5098fe71e826867034435f598b9"}
Oct 29 21:59:14.625707 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:14.625646    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podStartSLOduration=1.6256038080000002 podStartE2EDuration="1.625603808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:14.620292478 +0000 UTC m=+727.120334789" watchObservedRunningTime="2026-10-29 21:59:14.625603808 +0000 UTC m=+727.125646119"
Oct 29 21:59:15.036253 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:15.036213    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:15.547066 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:15.547030    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 29 21:59:15.547258 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:15.547086    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 29 21:59:16.546564 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:16.546529    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 29 21:59:16.546997 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:16.546602    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 29 21:59:17.546506 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:17.546426    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 29 21:59:17.546673 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:17.546497    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 29 21:59:18.551355 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:18.551322    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:18.551776 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:18.551717    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:18.552312 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:18.552292    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 21:59:24.035799 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.035767    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:24.049525 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.049502    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:24.703873 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.703845    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:24.851294 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.851266    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"]
Oct 29 21:59:24.855826 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.855807    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:24.870196 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.870153    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"]
Oct 29 21:59:24.882448 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.882411    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:24.975043 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.974972    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"]
Oct 29 21:59:24.982224 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.982198    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"]
Oct 29 21:59:24.982607 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.982578    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npmkq\" (UniqueName: \"kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-kube-api-access-npmkq\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:24.982711 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.982646    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th494\" (UniqueName: \"kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-kube-api-access-th494\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:24.982859 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.982716    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:24.982859 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.982747    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:24.982859 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.982787    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:24.982859 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.982819    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:24.983065 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.982887    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:24.983065 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.982926    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:24.983065 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.982951    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:24.983065 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.983007    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:24.983065 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.983040    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:24.983374 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.983349    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:24.983440 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.983424    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:24.983495 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.983467    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:24.983545 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.983530    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:24.983617 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.983602    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:24.983732 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.983672    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:24.983732 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:24.983715    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.085110 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085078    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.085574 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085118    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.085574 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085147    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.085574 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085222    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.085574 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085316    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.085574 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085351    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.085574 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085377    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085648    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085714    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085740    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085800    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085826    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085901    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085946    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.085959    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.086026    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.086045    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.086054    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.086125    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.086152    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.086159 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.086170    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.086902 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.086200    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.086902 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.086210    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.086902 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.086257    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npmkq\" (UniqueName: \"kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-kube-api-access-npmkq\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.086902 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.086286    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th494\" (UniqueName: \"kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-kube-api-access-th494\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.086902 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.086297    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.086902 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.086454    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.086902 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.086499    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.088338 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.088308    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.088454 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.088432    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.088900 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.088881    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.089038 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.089021    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.108095 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.108076    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.110835 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.110813    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th494\" (UniqueName: \"kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-kube-api-access-th494\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.111054 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.111034    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.111142 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.111125    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmkq\" (UniqueName: \"kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-kube-api-access-npmkq\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.271440 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.271368    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:25.293087 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.293061    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:25.603585 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.603548    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"]
Oct 29 21:59:25.605067 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:59:25.605040    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3379fba3_ce1e_4577_a370_ab3d675b4c4b.slice/crio-482bed1e652cf2c0decc67af645ef8a9b3d6610d3abeec83acd0cab1347ad776 WatchSource:0}: Error finding container 482bed1e652cf2c0decc67af645ef8a9b3d6610d3abeec83acd0cab1347ad776: Status 404 returned error can't find the container with id 482bed1e652cf2c0decc67af645ef8a9b3d6610d3abeec83acd0cab1347ad776
Oct 29 21:59:25.607729 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.607696    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:25.607796 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.607765    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:25.607829 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.607811    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:25.646717 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.646693    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"]
Oct 29 21:59:25.647321 ip-10-0-136-38 kubenswrapper[7082]: W1029 21:59:25.647296    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e47b26_da7b_4e93_aa45_16fc7cca65d0.slice/crio-13ea18c085d67134950166cffeed3bc7e237ae97054ccffb11d7c2980dfbb2d9 WatchSource:0}: Error finding container 13ea18c085d67134950166cffeed3bc7e237ae97054ccffb11d7c2980dfbb2d9: Status 404 returned error can't find the container with id 13ea18c085d67134950166cffeed3bc7e237ae97054ccffb11d7c2980dfbb2d9
Oct 29 21:59:25.649916 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.649885    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:25.650003 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.649946    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:25.650003 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.649973    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:25.704450 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.704195    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" event={"ID":"3379fba3-ce1e-4577-a370-ab3d675b4c4b","Type":"ContainerStarted","Data":"ff4adb076d831437ba08c8a7653fda6c51e2b9cac16123fa4f5fbd656878318f"}
Oct 29 21:59:25.704450 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.704282    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" event={"ID":"3379fba3-ce1e-4577-a370-ab3d675b4c4b","Type":"ContainerStarted","Data":"482bed1e652cf2c0decc67af645ef8a9b3d6610d3abeec83acd0cab1347ad776"}
Oct 29 21:59:25.706397 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.706368    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" event={"ID":"58e47b26-da7b-4e93-aa45-16fc7cca65d0","Type":"ContainerStarted","Data":"13ea18c085d67134950166cffeed3bc7e237ae97054ccffb11d7c2980dfbb2d9"}
Oct 29 21:59:25.759784 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:25.759720    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" podStartSLOduration=1.759701031 podStartE2EDuration="1.759701031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:25.755939712 +0000 UTC m=+738.255982023" watchObservedRunningTime="2026-10-29 21:59:25.759701031 +0000 UTC m=+738.259743331"
Oct 29 21:59:26.271822 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:26.271789    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:26.272994 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:26.272971    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 29 21:59:26.273107 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:26.273015    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 29 21:59:26.794654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:26.789298    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" event={"ID":"58e47b26-da7b-4e93-aa45-16fc7cca65d0","Type":"ContainerStarted","Data":"c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e"}
Oct 29 21:59:26.852651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:26.850289    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" podStartSLOduration=2.850266517 podStartE2EDuration="2.850266517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:26.849001372 +0000 UTC m=+739.349043673" watchObservedRunningTime="2026-10-29 21:59:26.850266517 +0000 UTC m=+739.350308815"
Oct 29 21:59:27.272309 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:27.272272    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 29 21:59:27.272754 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:27.272331    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 29 21:59:27.305651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:27.302410    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:27.305651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:27.303784    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 21:59:27.305651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:27.303849    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 21:59:28.280271 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:28.280236    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 29 21:59:28.280792 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:28.280301    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 29 21:59:28.294021 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:28.293993    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 21:59:28.294245 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:28.294221    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 21:59:28.673683 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:59:28.669941    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:59:29.271727 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:29.271692    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 29 21:59:29.272016 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:29.271745    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 29 21:59:29.312651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:29.309796    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 21:59:29.312651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:29.309850    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 21:59:30.280833 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:30.277053    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 29 21:59:30.280833 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:30.277137    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 29 21:59:30.298456 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:30.298423    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 21:59:30.298712 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:30.298685    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 21:59:31.273185 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:31.273146    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 29 21:59:31.273694 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:31.273210    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 29 21:59:31.316654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:31.315742    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 21:59:31.316654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:31.315867    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 21:59:32.272445 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:32.272413    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 29 21:59:32.272678 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:32.272470    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 29 21:59:32.293513 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:32.293481    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 29 21:59:32.294081 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:32.294051    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 29 21:59:33.308261 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:33.308203    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:33.310209 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:33.310185    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:33.311528 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:33.311506    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 21:59:33.441491 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:33.441452    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 21:59:33.655568 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:33.655492    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:33.655987 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:33.655963    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:33.657172 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:33.657153    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 21:59:37.053671 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:37.052780    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:37.594774 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:37.594727    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:37.729654 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:37.711981    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 21:59:38.015068 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.011771    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"]
Oct 29 21:59:38.057693 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.042376    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.057693 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.052943    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:38.141798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.071004    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:59:38.141798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.134658    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.141798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.134706    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.141798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.134739    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.141798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.134797    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hskn5\" (UniqueName: \"kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-kube-api-access-hskn5\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.141798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.134825    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50a168b-99ca-411d-b32a-19021ca6e029-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.141798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.134967    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50a168b-99ca-411d-b32a-19021ca6e029-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.141798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.135012    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.141798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.135156    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.141798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.135199    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.260427 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.238033    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"]
Oct 29 21:59:38.260427 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.238967    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50a168b-99ca-411d-b32a-19021ca6e029-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.260427 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.239013    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.260427 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.239129    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.260427 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.239198    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.260427 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.239274    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.260427 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.239300    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.260427 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.239330    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.260427 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.239383    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hskn5\" (UniqueName: \"kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-kube-api-access-hskn5\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.260427 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.239412    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50a168b-99ca-411d-b32a-19021ca6e029-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.260427 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.243608    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.260427 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.244341    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50a168b-99ca-411d-b32a-19021ca6e029-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.265716 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.262645    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.267213 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.266783    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.267213 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.267018    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.330947 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.330901    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:38.628229 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.628190    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 21:59:38.629143 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.629119    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:59:38.821712 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.796296    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 21:59:38.821712 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.796764    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 21:59:38.821712 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.820805    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:59:38.981717 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:38.981569    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 21:59:39.108649 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:39.099073    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:39.220661 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:39.219080    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hskn5\" (UniqueName: \"kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-kube-api-access-hskn5\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:39.244482 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:39.244207    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:39.265123 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:39.265070    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50a168b-99ca-411d-b32a-19021ca6e029-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:39.448105 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:39.448049    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:39.583210 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:39.583181    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:39.633804 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:39.633760    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:39.805717 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:39.802020    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:39.874703 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:39.868319    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:39.882961 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:39.882915    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:40.021502 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:40.021465    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:40.030655 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:40.025358    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 21:59:40.030655 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:40.026136    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:59:40.103827 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:40.103778    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:40.638660 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:40.634715    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:40.812653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:40.808234    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 21:59:40.899489 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:40.899392    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:41.029242 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:41.028218    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 21:59:41.109746 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:41.109151    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:41.670716 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:41.657473    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 21:59:41.809713 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:41.809670    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:41.994647 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:41.980957    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:42.059292 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:42.018599    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:42.059292 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:42.018712    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:42.059292 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:42.018758    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:42.059292 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:42.030008    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 21:59:42.125712 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:42.115067    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 21:59:42.171476 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:42.171165    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:42.365869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:42.365835    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"]
Oct 29 21:59:42.660841 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:42.660741    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:42.810860 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:42.810796    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:42.852666 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:42.852604    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" event={"ID":"f50a168b-99ca-411d-b32a-19021ca6e029","Type":"ContainerStarted","Data":"88dfab60c76ff24fc90b752b402d4efa7bac9f4d126b1c42f093730fff478205"}
Oct 29 21:59:42.997815 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:42.996697    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 21:59:43.022135 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.022104    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 21:59:43.031785 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.031747    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 21:59:43.040740 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.040703    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 21:59:43.041000 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.040971    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:43.163798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.147125    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 21:59:43.163798 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.147190    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:43.260869 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.247587    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 21:59:43.367653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.345109    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 21:59:43.367653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.345164    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:43.384665 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.372275    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 21:59:43.384665 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.373800    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 21:59:43.714653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.713234    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 21:59:43.865294 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.865248    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" event={"ID":"f50a168b-99ca-411d-b32a-19021ca6e029","Type":"ContainerStarted","Data":"99eaa8dd4a77b1b75062415752e29ee672ad20cb2cdbcf1c6bd9bb3bdf2085b5"}
Oct 29 21:59:43.916499 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:43.916390    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" podStartSLOduration=6.916370411 podStartE2EDuration="6.916370411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:43.915443562 +0000 UTC m=+756.415485856" watchObservedRunningTime="2026-10-29 21:59:43.916370411 +0000 UTC m=+756.416412711"
Oct 29 21:59:44.597653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:44.591899    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:44.597653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:44.592085    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 21:59:44.597653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:44.592131    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 21:59:45.592652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:45.589971    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 21:59:45.592652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:45.590035    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 21:59:46.477766 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.477737    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"]
Oct 29 21:59:46.482638 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.482606    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.523671 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.517765    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8a9e5f\"/\"openshift-service-ca.crt\""
Oct 29 21:59:46.523671 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.517875    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8a9e5f\"/\"kube-root-ca.crt\""
Oct 29 21:59:46.523671 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.517979    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a9e5f\"/\"default-dockercfg-q247w\""
Oct 29 21:59:46.523671 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.518091    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a9e5f\"/\"distinct-2-w3-8a9e5f-kserve-self-signed-certs\""
Oct 29 21:59:46.532872 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.532848    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"]
Oct 29 21:59:46.562186 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.562158    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-dshm\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.562338 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.562206    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6fbl\" (UniqueName: \"kubernetes.io/projected/55aac7b4-d385-4ba9-b5a0-7528b481f142-kube-api-access-s6fbl\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.562338 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.562270    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55aac7b4-d385-4ba9-b5a0-7528b481f142-tls-certs\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.562425 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.562353    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-home\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.562425 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.562385    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-tmp-dir\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.562425 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.562421    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-model-cache\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.605650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.599969    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 21:59:46.605650 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.600024    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 21:59:46.680189 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.680149    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-home\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.680384 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.680231    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-tmp-dir\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.680384 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.680300    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-model-cache\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.680506 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.680405    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-dshm\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.680506 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.680459    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6fbl\" (UniqueName: \"kubernetes.io/projected/55aac7b4-d385-4ba9-b5a0-7528b481f142-kube-api-access-s6fbl\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.680620 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.680575    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55aac7b4-d385-4ba9-b5a0-7528b481f142-tls-certs\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.693653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.688079    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-model-cache\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.693653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.690936    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-tmp-dir\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.693653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.691149    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-home\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.712679 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.710685    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55aac7b4-d385-4ba9-b5a0-7528b481f142-tls-certs\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.749799 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.743757    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-dshm\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.796662 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.783088    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6fbl\" (UniqueName: \"kubernetes.io/projected/55aac7b4-d385-4ba9-b5a0-7528b481f142-kube-api-access-s6fbl\") pod \"distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:46.840088 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:46.840045    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:47.225947 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:47.225784    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"]
Oct 29 21:59:47.592652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:47.589932    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 21:59:47.592652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:47.589985    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 21:59:47.959840 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:47.959447    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" event={"ID":"55aac7b4-d385-4ba9-b5a0-7528b481f142","Type":"ContainerStarted","Data":"357d2a0fc2fdb8071a7f13d320d9385b14577655eab7ee9060441f1ab8e38440"}
Oct 29 21:59:47.959840 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:47.959503    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" event={"ID":"55aac7b4-d385-4ba9-b5a0-7528b481f142","Type":"ContainerStarted","Data":"8731253ca7a95d5e381b21e5db9813f00821338bf0cfee0208ad374a9122ae2d"}
Oct 29 21:59:48.015652 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:48.015119    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podStartSLOduration=2.015100487 podStartE2EDuration="2.015100487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:48.010049297 +0000 UTC m=+760.510091599" watchObservedRunningTime="2026-10-29 21:59:48.015100487 +0000 UTC m=+760.515142787"
Oct 29 21:59:48.594651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:48.589972    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 21:59:48.594651 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:48.590036    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 21:59:48.841550 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:48.841523    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:49.595653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:49.589005    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 21:59:49.595653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:49.589065    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 21:59:50.587570 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:50.587535    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 29 21:59:50.587761 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:50.587596    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 29 21:59:51.591428 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:51.591399    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:51.591902 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:51.591878    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:51.592678 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:51.592656    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 21:59:53.208522 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.208488    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"]
Oct 29 21:59:53.214444 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.214422    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.250263 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.245705    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"]
Oct 29 21:59:53.258769 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.258741    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.258944 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.258919    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.259011 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.258976    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.259065 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.259037    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.259118 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.259096    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.259229 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.259213    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.259284 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.259245    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.259284 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.259274    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.259380 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.259295    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m94hd\" (UniqueName: \"kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-kube-api-access-m94hd\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.360922 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.360888    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.361111 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.360930    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.361111 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.360973    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.361111 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.361009    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.361286 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.361116    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.361286 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.361147    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.361286 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.361187    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.361286 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.361210    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m94hd\" (UniqueName: \"kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-kube-api-access-m94hd\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.361487 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.361354    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.361487 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.361383    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.361487 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.361412    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.361619 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.361556    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.361703 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.361687    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.361913 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.361885    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.363742 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.363722    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.363742 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.363736    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.384802 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.384773    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.385365 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.385343    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m94hd\" (UniqueName: \"kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-kube-api-access-m94hd\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.535819 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.535731    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:53.875838 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.875180    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"]
Oct 29 21:59:53.881782 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.881727    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:53.881902 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.881872    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:53.881990 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:53.881968    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:54.055696 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.055656    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" event={"ID":"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38","Type":"ContainerStarted","Data":"e8f11674dc4876ad34473b84353bcee1d2af4b7ad0641a8074cb593b5ec91c45"}
Oct 29 21:59:54.055885 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.055724    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" event={"ID":"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38","Type":"ContainerStarted","Data":"83df29759509283d88aae061b012f0c33ad41a97a72ad6b3e52784cf2d4cf5f8"}
Oct 29 21:59:54.108733 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.108644    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podStartSLOduration=1.108606496 podStartE2EDuration="1.108606496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:54.102567825 +0000 UTC m=+766.602610126" watchObservedRunningTime="2026-10-29 21:59:54.108606496 +0000 UTC m=+766.608648791"
Oct 29 21:59:54.233075 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.232982    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"]
Oct 29 21:59:54.238482 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.238446    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.263329 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.263298    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"]
Oct 29 21:59:54.274485 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.274456    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.274619 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.274573    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.274694 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.274677    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85ac047-ae41-4a6d-9711-87e13cf1917b-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.274819 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.274787    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf488\" (UniqueName: \"kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-kube-api-access-cf488\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.274914 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.274889    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.275030 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.275001    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.275142 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.275123    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.275234 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.275221    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.275288 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.275251    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.377055 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.377018    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.377230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.377120    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.377230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.377174    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85ac047-ae41-4a6d-9711-87e13cf1917b-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.377230 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.377205    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf488\" (UniqueName: \"kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-kube-api-access-cf488\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.377382 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.377233    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.377382 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.377266    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.377482 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.377422    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.377555 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.377530    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.377611 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.377539    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.377611 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.377603    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.377863 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.377841    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.378016 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.377989    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85ac047-ae41-4a6d-9711-87e13cf1917b-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.378136 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.378000    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.378243 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.378222    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.379675 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.379649    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.380345 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.380318    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.400846 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.400821    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.400970 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.400912    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf488\" (UniqueName: \"kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-kube-api-access-cf488\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:54.536723 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.536636    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 21:59:54.537865 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.537837    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 21:59:54.538005 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.537891    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 21:59:54.578719 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:54.578601    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:55.024121 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:55.024091    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"]
Oct 29 21:59:55.026878 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:55.026837    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:55.026962 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:55.026926    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:55.027007 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:55.026972    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 21:59:55.064095 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:55.064065    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" event={"ID":"f85ac047-ae41-4a6d-9711-87e13cf1917b","Type":"ContainerStarted","Data":"f8cf057c17adaa37afc48334e1c36d8e22cffda1a2df70e45842bd19f01943d8"}
Oct 29 21:59:55.536963 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:55.536930    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 21:59:55.537377 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:55.536990    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 21:59:56.074781 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:56.074744    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" event={"ID":"f85ac047-ae41-4a6d-9711-87e13cf1917b","Type":"ContainerStarted","Data":"4608140203d245d18679ad7e6432b15765ed1c10bb2a4a0fc92d01aca22ae25c"}
Oct 29 21:59:56.149653 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:56.149323    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" podStartSLOduration=2.149302819 podStartE2EDuration="2.149302819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:56.14613008 +0000 UTC m=+768.646172380" watchObservedRunningTime="2026-10-29 21:59:56.149302819 +0000 UTC m=+768.649345120"
Oct 29 21:59:56.546710 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:56.544574    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 21:59:56.546710 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:56.544666    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 21:59:56.579058 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:56.579031    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 21:59:56.580128 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:56.580105    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 21:59:56.580309 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:56.580291    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 21:59:56.841655 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:56.841009    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:56.974241 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:56.974206    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:57.158754 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:57.158682    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 21:59:57.546707 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:57.541731    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 21:59:57.546707 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:57.541798    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 21:59:57.579020 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:57.578987    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 21:59:57.579188 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:57.579032    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 21:59:58.467033 ip-10-0-136-38 kubenswrapper[7082]: E1029 21:59:58.466995    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 21:59:58.538418 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:58.537987    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 21:59:58.538418 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:58.538046    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 21:59:58.579057 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:58.579013    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 21:59:58.579612 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:58.579541    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 21:59:59.542059 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:59.542020    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 21:59:59.542312 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:59.542280    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 21:59:59.579160 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:59.579131    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 21:59:59.579529 ip-10-0-136-38 kubenswrapper[7082]: I1029 21:59:59.579181    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:00:00.167515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.167474    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"]
Oct 29 22:00:00.173421 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.173394    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"]
Oct 29 22:00:00.173546 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.173516    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"
Oct 29 22:00:00.180851 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.180825    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"]
Oct 29 22:00:00.184068 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.184032    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"
Oct 29 22:00:00.195745 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.193186    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:00:00.202668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.197798    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:00:00.202668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.197860    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:00:00.337418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.337379    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh\" (UID: \"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"
Oct 29 22:00:00.337773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.337747    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb8hw\" (UniqueName: \"kubernetes.io/projected/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-kube-api-access-qb8hw\") pod \"maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8\" (UID: \"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"
Oct 29 22:00:00.360921 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.360883    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6prp\" (UniqueName: \"kubernetes.io/projected/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-kube-api-access-q6prp\") pod \"maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh\" (UID: \"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"
Oct 29 22:00:00.361200 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.361176    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8\" (UID: \"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"
Oct 29 22:00:00.371020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.370993    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"]
Oct 29 22:00:00.373706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.372819    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"
Oct 29 22:00:00.437722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.434870    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"]
Oct 29 22:00:00.462600 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.461443    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"
Oct 29 22:00:00.483694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.483646    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e897f3-9840-42c8-8f21-b6ce623ef443-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888520-gfcjc\" (UID: \"b3e897f3-9840-42c8-8f21-b6ce623ef443\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"
Oct 29 22:00:00.483909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.483882    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6prp\" (UniqueName: \"kubernetes.io/projected/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-kube-api-access-q6prp\") pod \"maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh\" (UID: \"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"
Oct 29 22:00:00.484076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.484044    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg6vw\" (UniqueName: \"kubernetes.io/projected/b3e897f3-9840-42c8-8f21-b6ce623ef443-kube-api-access-sg6vw\") pod \"maas-api-key-cleanup-29888520-gfcjc\" (UID: \"b3e897f3-9840-42c8-8f21-b6ce623ef443\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"
Oct 29 22:00:00.484500 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.484474    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8\" (UID: \"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"
Oct 29 22:00:00.488645 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.484749    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh\" (UID: \"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"
Oct 29 22:00:00.525648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.506901    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb8hw\" (UniqueName: \"kubernetes.io/projected/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-kube-api-access-qb8hw\") pod \"maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8\" (UID: \"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"
Oct 29 22:00:00.543700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.487262    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8\" (UID: \"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"
Oct 29 22:00:00.543700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.542120    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"]
Oct 29 22:00:00.543700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.542225    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:00:00.543700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.543410    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 22:00:00.544699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.543131    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6prp\" (UniqueName: \"kubernetes.io/projected/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-kube-api-access-q6prp\") pod \"maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh\" (UID: \"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"
Oct 29 22:00:00.544699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.544605    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh\" (UID: \"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"
Oct 29 22:00:00.546570 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.546545    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"
Oct 29 22:00:00.551297 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.548347    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"]
Oct 29 22:00:00.551297 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.548394    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"]
Oct 29 22:00:00.552039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.551991    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"]
Oct 29 22:00:00.556118 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.556091    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"]
Oct 29 22:00:00.563976 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.563955    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"]
Oct 29 22:00:00.565663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.565604    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb8hw\" (UniqueName: \"kubernetes.io/projected/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-kube-api-access-qb8hw\") pod \"maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8\" (UID: \"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"
Oct 29 22:00:00.569503 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.569484    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"]
Oct 29 22:00:00.569668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.569620    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.579311 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.579288    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:00:00.579610 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.579338    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:00:00.651690 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.651657    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e897f3-9840-42c8-8f21-b6ce623ef443-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888520-gfcjc\" (UID: \"b3e897f3-9840-42c8-8f21-b6ce623ef443\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"
Oct 29 22:00:00.651834 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.651705    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.651834 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.651761    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84310bdd-05f6-4107-9430-68fecce3fcec-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.651834 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.651789    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.651982 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.651847    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg6vw\" (UniqueName: \"kubernetes.io/projected/b3e897f3-9840-42c8-8f21-b6ce623ef443-kube-api-access-sg6vw\") pod \"maas-api-key-cleanup-29888520-gfcjc\" (UID: \"b3e897f3-9840-42c8-8f21-b6ce623ef443\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"
Oct 29 22:00:00.652087 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.652045    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.652159 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.652106    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.652217 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.652161    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4zb5\" (UniqueName: \"kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-kube-api-access-g4zb5\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.652271 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.652253    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd3c41-6238-4a00-af83-becbfaef0e0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv\" (UID: \"0fcd3c41-6238-4a00-af83-becbfaef0e0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"
Oct 29 22:00:00.652324 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.652278    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.652324 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.652301    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84310bdd-05f6-4107-9430-68fecce3fcec-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.652442 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.652336    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.652499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.652458    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59ddz\" (UniqueName: \"kubernetes.io/projected/051ea159-740b-4cc4-a5ef-1e9b03945cc0-kube-api-access-59ddz\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w\" (UID: \"051ea159-740b-4cc4-a5ef-1e9b03945cc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"
Oct 29 22:00:00.652578 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.652555    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e897f3-9840-42c8-8f21-b6ce623ef443-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888520-gfcjc\" (UID: \"b3e897f3-9840-42c8-8f21-b6ce623ef443\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"
Oct 29 22:00:00.652803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.652774    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/051ea159-740b-4cc4-a5ef-1e9b03945cc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w\" (UID: \"051ea159-740b-4cc4-a5ef-1e9b03945cc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"
Oct 29 22:00:00.652899 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.652833    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjv8n\" (UniqueName: \"kubernetes.io/projected/0fcd3c41-6238-4a00-af83-becbfaef0e0b-kube-api-access-cjv8n\") pod \"maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv\" (UID: \"0fcd3c41-6238-4a00-af83-becbfaef0e0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"
Oct 29 22:00:00.671410 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.671385    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"
Oct 29 22:00:00.674964 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.674693    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg6vw\" (UniqueName: \"kubernetes.io/projected/b3e897f3-9840-42c8-8f21-b6ce623ef443-kube-api-access-sg6vw\") pod \"maas-api-key-cleanup-29888520-gfcjc\" (UID: \"b3e897f3-9840-42c8-8f21-b6ce623ef443\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"
Oct 29 22:00:00.682717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.682696    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.753925    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/051ea159-740b-4cc4-a5ef-1e9b03945cc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w\" (UID: \"051ea159-740b-4cc4-a5ef-1e9b03945cc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.753987    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjv8n\" (UniqueName: \"kubernetes.io/projected/0fcd3c41-6238-4a00-af83-becbfaef0e0b-kube-api-access-cjv8n\") pod \"maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv\" (UID: \"0fcd3c41-6238-4a00-af83-becbfaef0e0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.754060    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.754113    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84310bdd-05f6-4107-9430-68fecce3fcec-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.754140    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.754192    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.754220    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.754258    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4zb5\" (UniqueName: \"kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-kube-api-access-g4zb5\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.754322    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd3c41-6238-4a00-af83-becbfaef0e0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv\" (UID: \"0fcd3c41-6238-4a00-af83-becbfaef0e0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.754347    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.754373    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84310bdd-05f6-4107-9430-68fecce3fcec-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.754408    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.754479    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59ddz\" (UniqueName: \"kubernetes.io/projected/051ea159-740b-4cc4-a5ef-1e9b03945cc0-kube-api-access-59ddz\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w\" (UID: \"051ea159-740b-4cc4-a5ef-1e9b03945cc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"
Oct 29 22:00:00.755043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.754941    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.756949 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.755578    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/051ea159-740b-4cc4-a5ef-1e9b03945cc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w\" (UID: \"051ea159-740b-4cc4-a5ef-1e9b03945cc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"
Oct 29 22:00:00.756949 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.756070    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.756949 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.756293    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84310bdd-05f6-4107-9430-68fecce3fcec-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.756949 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.756474    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd3c41-6238-4a00-af83-becbfaef0e0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv\" (UID: \"0fcd3c41-6238-4a00-af83-becbfaef0e0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"
Oct 29 22:00:00.756949 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.756534    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.757514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.757478    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.759543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.759525    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.759868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.759847    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84310bdd-05f6-4107-9430-68fecce3fcec-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.779194 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.779128    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59ddz\" (UniqueName: \"kubernetes.io/projected/051ea159-740b-4cc4-a5ef-1e9b03945cc0-kube-api-access-59ddz\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w\" (UID: \"051ea159-740b-4cc4-a5ef-1e9b03945cc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"
Oct 29 22:00:00.779371 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.779326    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4zb5\" (UniqueName: \"kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-kube-api-access-g4zb5\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.780074 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.779810    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:00.781924 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.781899    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjv8n\" (UniqueName: \"kubernetes.io/projected/0fcd3c41-6238-4a00-af83-becbfaef0e0b-kube-api-access-cjv8n\") pod \"maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv\" (UID: \"0fcd3c41-6238-4a00-af83-becbfaef0e0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"
Oct 29 22:00:00.853836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.853504    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"
Oct 29 22:00:00.857966 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.857906    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"
Oct 29 22:00:00.937671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.937607    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"
Oct 29 22:00:00.945404 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:00.945299    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:01.149643 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:00:01.149585    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f3f1b3_42aa_469c_a8e5_8360c5bcd18f.slice/crio-9d8b66c7ac9a4448ece1a03603670f8f938cc39565c27f4ecb299c47b9c8a1ab WatchSource:0}: Error finding container 9d8b66c7ac9a4448ece1a03603670f8f938cc39565c27f4ecb299c47b9c8a1ab: Status 404 returned error can't find the container with id 9d8b66c7ac9a4448ece1a03603670f8f938cc39565c27f4ecb299c47b9c8a1ab
Oct 29 22:00:01.161366 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:01.156731    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"]
Oct 29 22:00:01.169897 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:01.161843    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"]
Oct 29 22:00:01.348215 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:01.348178    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh" event={"ID":"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c","Type":"ContainerStarted","Data":"795d185662eaffb31f6ea16216dcab02ed93202372e8f56f9bc20288b58ad0f1"}
Oct 29 22:00:01.350943 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:01.350869    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8" event={"ID":"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f","Type":"ContainerStarted","Data":"9d8b66c7ac9a4448ece1a03603670f8f938cc39565c27f4ecb299c47b9c8a1ab"}
Oct 29 22:00:01.558706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:01.540478    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:00:01.558706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:01.540542    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 29 22:00:01.625689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:01.617865    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:00:01.625689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:01.617925    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:00:01.639167 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:01.639138    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"]
Oct 29 22:00:01.885776 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:01.841804    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"]
Oct 29 22:00:02.102930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.102354    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"]
Oct 29 22:00:02.206810 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.205746    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:02.206810 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.205839    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:02.206810 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.205877    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:02.232749 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.232721    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"]
Oct 29 22:00:02.483128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.483048    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv" event={"ID":"0fcd3c41-6238-4a00-af83-becbfaef0e0b","Type":"ContainerStarted","Data":"3e8c207a57d9db020b0d16a5aa210cd762d61dc0c224ff4dd5155f55d3d5a651"}
Oct 29 22:00:02.484779 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.484750    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" event={"ID":"84310bdd-05f6-4107-9430-68fecce3fcec","Type":"ContainerStarted","Data":"a5121ad22562f5911dd37c45fbdffde1d7f2506a5fae99c45dc062dedf1ed408"}
Oct 29 22:00:02.487831 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.487789    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w" event={"ID":"051ea159-740b-4cc4-a5ef-1e9b03945cc0","Type":"ContainerStarted","Data":"5dae207ea4975838880f9636f7a54c701813d38a51f1b68745b2d98fd2149ba2"}
Oct 29 22:00:02.507011 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.506981    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh" event={"ID":"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c","Type":"ContainerStarted","Data":"df88969a3ddbbfd4a8c9b383b75f6fbb79bb75b3eb220d938a3f6425c418434d"}
Oct 29 22:00:02.525741 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.524400    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8" event={"ID":"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f","Type":"ContainerStarted","Data":"0bd8b70ac6d944545e265d4abf9e447e7dea7ee7914533080245c0df82bc4dbd"}
Oct 29 22:00:02.529980 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.529934    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc" event={"ID":"b3e897f3-9840-42c8-8f21-b6ce623ef443","Type":"ContainerStarted","Data":"f01cda640bb71991ed03a8635d0e84bebb6be59715c015e503f6e87d47acfcf5"}
Oct 29 22:00:02.560977 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.560949    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 22:00:02.562619 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.562601    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 22:00:02.578810 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.578787    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 22:00:02.580999 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.580934    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:00:02.581187 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.581165    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 29 22:00:02.581761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.581718    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh" podStartSLOduration=2.581701535 podStartE2EDuration="2.581701535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:02.577117374 +0000 UTC m=+775.077159674" watchObservedRunningTime="2026-10-29 22:00:02.581701535 +0000 UTC m=+775.081743833"
Oct 29 22:00:02.784283 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:02.784219    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8" podStartSLOduration=2.78419687 podStartE2EDuration="2.78419687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:02.693657818 +0000 UTC m=+775.193700118" watchObservedRunningTime="2026-10-29 22:00:02.78419687 +0000 UTC m=+775.284239170"
Oct 29 22:00:03.622898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.613878    7082 generic.go:356] "Generic (PLEG): container finished" podID="b3e897f3-9840-42c8-8f21-b6ce623ef443" containerID="e7fed022052f37bfb1fa387f53a7f182ababe33b6b2ea19492bb28d8adc48e9a" exitCode=60
Oct 29 22:00:03.622898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.614071    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc" event={"ID":"b3e897f3-9840-42c8-8f21-b6ce623ef443","Type":"ContainerDied","Data":"e7fed022052f37bfb1fa387f53a7f182ababe33b6b2ea19492bb28d8adc48e9a"}
Oct 29 22:00:03.622898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.614562    7082 scope.go:122] "RemoveContainer" containerID="e7fed022052f37bfb1fa387f53a7f182ababe33b6b2ea19492bb28d8adc48e9a"
Oct 29 22:00:03.647526 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.647409    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv" event={"ID":"0fcd3c41-6238-4a00-af83-becbfaef0e0b","Type":"ContainerStarted","Data":"68502db9d6b4788598589a6564543ad31ed3b130d3b699147af262112afbbc96"}
Oct 29 22:00:03.657345 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.657263    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" event={"ID":"84310bdd-05f6-4107-9430-68fecce3fcec","Type":"ContainerStarted","Data":"7809d0bf655c2143cfaa371b2b79452a5eded8cc81c5f1a54f295a560aeb575e"}
Oct 29 22:00:03.671548 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.671522    7082 generic.go:356] "Generic (PLEG): container finished" podID="051ea159-740b-4cc4-a5ef-1e9b03945cc0" containerID="f79dbecacccfc8abbd5b65084d30d2e030fff4428180a7a474f82700fac89bfb" exitCode=60
Oct 29 22:00:03.671817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.671791    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w" event={"ID":"051ea159-740b-4cc4-a5ef-1e9b03945cc0","Type":"ContainerDied","Data":"f79dbecacccfc8abbd5b65084d30d2e030fff4428180a7a474f82700fac89bfb"}
Oct 29 22:00:03.671936 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.671914    7082 scope.go:122] "RemoveContainer" containerID="f79dbecacccfc8abbd5b65084d30d2e030fff4428180a7a474f82700fac89bfb"
Oct 29 22:00:03.684878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.683970    7082 generic.go:356] "Generic (PLEG): container finished" podID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" containerID="df88969a3ddbbfd4a8c9b383b75f6fbb79bb75b3eb220d938a3f6425c418434d" exitCode=60
Oct 29 22:00:03.684878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.684044    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh" event={"ID":"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c","Type":"ContainerDied","Data":"df88969a3ddbbfd4a8c9b383b75f6fbb79bb75b3eb220d938a3f6425c418434d"}
Oct 29 22:00:03.684878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.684376    7082 scope.go:122] "RemoveContainer" containerID="df88969a3ddbbfd4a8c9b383b75f6fbb79bb75b3eb220d938a3f6425c418434d"
Oct 29 22:00:03.742302 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.742257    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 22:00:03.743413 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.743302    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 22:00:03.747144 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.747066    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv" podStartSLOduration=3.747052708 podStartE2EDuration="3.747052708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:03.743839767 +0000 UTC m=+776.243882067" watchObservedRunningTime="2026-10-29 22:00:03.747052708 +0000 UTC m=+776.247095044"
Oct 29 22:00:03.762205 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.762168    7082 generic.go:356] "Generic (PLEG): container finished" podID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" containerID="0bd8b70ac6d944545e265d4abf9e447e7dea7ee7914533080245c0df82bc4dbd" exitCode=60
Oct 29 22:00:03.762306 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.762222    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 22:00:03.762306 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.762275    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8" event={"ID":"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f","Type":"ContainerDied","Data":"0bd8b70ac6d944545e265d4abf9e447e7dea7ee7914533080245c0df82bc4dbd"}
Oct 29 22:00:03.763034 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.763010    7082 scope.go:122] "RemoveContainer" containerID="0bd8b70ac6d944545e265d4abf9e447e7dea7ee7914533080245c0df82bc4dbd"
Oct 29 22:00:03.814251 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.812512    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podStartSLOduration=3.812493873 podStartE2EDuration="3.812493873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:03.811811733 +0000 UTC m=+776.311854069" watchObservedRunningTime="2026-10-29 22:00:03.812493873 +0000 UTC m=+776.312536172"
Oct 29 22:00:03.946130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.946088    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:03.947091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.947050    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:00:03.947235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:03.947105    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 29 22:00:04.179180 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.179114    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"]
Oct 29 22:00:04.183589 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.183569    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.245470 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.244799    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"]
Oct 29 22:00:04.343384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.343356    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mrv9\" (UniqueName: \"kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-kube-api-access-9mrv9\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.343492 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.343404    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ac5793-4f48-4d6a-8390-19de6f99a814-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.343542 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.343487    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.343542 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.343516    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.343665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.343542    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.343718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.343686    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.343864 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.343844    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.344003 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.343988    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.344068 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.344045    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.446117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.446029    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mrv9\" (UniqueName: \"kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-kube-api-access-9mrv9\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.446117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.446074    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ac5793-4f48-4d6a-8390-19de6f99a814-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.446117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.446102    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.446398 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.446231    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.446398 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.446290    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.446398 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.446360    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.446542 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.446503    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.446701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.446673    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.446902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.446702    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.447018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.446918    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ac5793-4f48-4d6a-8390-19de6f99a814-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.447018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.446958    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.447182 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.447160    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.447246 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.447188    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.447412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.447383    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.448774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.448754    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.449294 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.449269    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.469289 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.469264    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.469415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.469397    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mrv9\" (UniqueName: \"kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-kube-api-access-9mrv9\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.547564 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.547529    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:04.772233 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.772147    7082 generic.go:356] "Generic (PLEG): container finished" podID="0fcd3c41-6238-4a00-af83-becbfaef0e0b" containerID="68502db9d6b4788598589a6564543ad31ed3b130d3b699147af262112afbbc96" exitCode=60
Oct 29 22:00:04.772383 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.772225    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv" event={"ID":"0fcd3c41-6238-4a00-af83-becbfaef0e0b","Type":"ContainerDied","Data":"68502db9d6b4788598589a6564543ad31ed3b130d3b699147af262112afbbc96"}
Oct 29 22:00:04.772559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.772538    7082 scope.go:122] "RemoveContainer" containerID="68502db9d6b4788598589a6564543ad31ed3b130d3b699147af262112afbbc96"
Oct 29 22:00:04.778889 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.778853    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w" event={"ID":"051ea159-740b-4cc4-a5ef-1e9b03945cc0","Type":"ContainerStarted","Data":"44c5fc77fafc81f1863acb8174578c91d9ae4bc2a019585047a81c772412fa1f"}
Oct 29 22:00:04.781863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.781164    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh" event={"ID":"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c","Type":"ContainerStarted","Data":"96bdb2cd8a7b18d5cd4df5f3c4865e0852ce1fb0fb36bb28cb99f240d27120df"}
Oct 29 22:00:04.784120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.784091    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8" event={"ID":"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f","Type":"ContainerStarted","Data":"6169044f24642d6c75cc04f7d31706c52e923e1bd4da907efaaf6a65328c3daf"}
Oct 29 22:00:04.817656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.810720    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc" event={"ID":"b3e897f3-9840-42c8-8f21-b6ce623ef443","Type":"ContainerStarted","Data":"ece7daf002431675170cc62a2cbd7107d12d5b143b23081238adc818134a56d9"}
Oct 29 22:00:04.869846 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.869802    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc" podStartSLOduration=4.869786913 podStartE2EDuration="4.869786913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:04.865951473 +0000 UTC m=+777.365993769" watchObservedRunningTime="2026-10-29 22:00:04.869786913 +0000 UTC m=+777.369829211"
Oct 29 22:00:04.940580 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.940075    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:04.940580 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.940161    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:04.940580 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.940208    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:04.942782 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.942757    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"]
Oct 29 22:00:04.946172 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.946144    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:00:04.946268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.946194    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 29 22:00:04.969439 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:04.969399    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w" podStartSLOduration=4.969386382 podStartE2EDuration="4.969386382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:04.967038766 +0000 UTC m=+777.467081066" watchObservedRunningTime="2026-10-29 22:00:04.969386382 +0000 UTC m=+777.469428681"
Oct 29 22:00:05.436649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.435779    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"]
Oct 29 22:00:05.572653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.572554    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"]
Oct 29 22:00:05.572899 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.572794    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.615514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.601730    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whqf\" (UniqueName: \"kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-kube-api-access-2whqf\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.615514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.608889    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.615514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.609018    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.615514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.609081    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.615514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.609197    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694c4fbc-01cd-4141-8168-2a4c261e0103-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.615514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.609308    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.615514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.609374    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.615514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.609530    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.615514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.609680    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.720757 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.720443    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.720974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.720913    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.721337 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.721142    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694c4fbc-01cd-4141-8168-2a4c261e0103-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.722742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.721775    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.722742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.721835    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.722742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.721959    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.722742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.722078    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.722742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.722175    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whqf\" (UniqueName: \"kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-kube-api-access-2whqf\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.722742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.722183    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.722742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.722205    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.722742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.722283    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694c4fbc-01cd-4141-8168-2a4c261e0103-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.722742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.722448    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.722742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.722556    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.723493 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.723467    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.725359 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.725339    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.725732 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.725713    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.748418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.748399    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whqf\" (UniqueName: \"kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-kube-api-access-2whqf\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.750092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.750060    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.827823 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.827742    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv" event={"ID":"0fcd3c41-6238-4a00-af83-becbfaef0e0b","Type":"ContainerStarted","Data":"8e04e9429113bb3769e80dc4433ff815b0eb26282853a156937a50d6e628eebf"}
Oct 29 22:00:05.829972 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.829947    7082 generic.go:356] "Generic (PLEG): container finished" podID="051ea159-740b-4cc4-a5ef-1e9b03945cc0" containerID="44c5fc77fafc81f1863acb8174578c91d9ae4bc2a019585047a81c772412fa1f" exitCode=60
Oct 29 22:00:05.830114 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.830003    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w" event={"ID":"051ea159-740b-4cc4-a5ef-1e9b03945cc0","Type":"ContainerDied","Data":"44c5fc77fafc81f1863acb8174578c91d9ae4bc2a019585047a81c772412fa1f"}
Oct 29 22:00:05.830114 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.830060    7082 scope.go:122] "RemoveContainer" containerID="f79dbecacccfc8abbd5b65084d30d2e030fff4428180a7a474f82700fac89bfb"
Oct 29 22:00:05.830372 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.830355    7082 scope.go:122] "RemoveContainer" containerID="44c5fc77fafc81f1863acb8174578c91d9ae4bc2a019585047a81c772412fa1f"
Oct 29 22:00:05.830737 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:05.830698    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w_odh-ai-gateway-infra(051ea159-740b-4cc4-a5ef-1e9b03945cc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w" podUID="051ea159-740b-4cc4-a5ef-1e9b03945cc0"
Oct 29 22:00:05.832424 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.832403    7082 generic.go:356] "Generic (PLEG): container finished" podID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" containerID="96bdb2cd8a7b18d5cd4df5f3c4865e0852ce1fb0fb36bb28cb99f240d27120df" exitCode=60
Oct 29 22:00:05.832510 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.832430    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh" event={"ID":"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c","Type":"ContainerDied","Data":"96bdb2cd8a7b18d5cd4df5f3c4865e0852ce1fb0fb36bb28cb99f240d27120df"}
Oct 29 22:00:05.832766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.832745    7082 scope.go:122] "RemoveContainer" containerID="96bdb2cd8a7b18d5cd4df5f3c4865e0852ce1fb0fb36bb28cb99f240d27120df"
Oct 29 22:00:05.833131 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:05.833107    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh_odh-ai-gateway-infra(23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh" podUID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c"
Oct 29 22:00:05.834553 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.834533    7082 generic.go:356] "Generic (PLEG): container finished" podID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" containerID="6169044f24642d6c75cc04f7d31706c52e923e1bd4da907efaaf6a65328c3daf" exitCode=60
Oct 29 22:00:05.834653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.834554    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8" event={"ID":"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f","Type":"ContainerDied","Data":"6169044f24642d6c75cc04f7d31706c52e923e1bd4da907efaaf6a65328c3daf"}
Oct 29 22:00:05.834828 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.834814    7082 scope.go:122] "RemoveContainer" containerID="6169044f24642d6c75cc04f7d31706c52e923e1bd4da907efaaf6a65328c3daf"
Oct 29 22:00:05.835076 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:05.835045    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8_odh-ai-gateway-infra(e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f"
Oct 29 22:00:05.837965 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.837455    7082 generic.go:356] "Generic (PLEG): container finished" podID="b3e897f3-9840-42c8-8f21-b6ce623ef443" containerID="ece7daf002431675170cc62a2cbd7107d12d5b143b23081238adc818134a56d9" exitCode=60
Oct 29 22:00:05.837965 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.837494    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc" event={"ID":"b3e897f3-9840-42c8-8f21-b6ce623ef443","Type":"ContainerDied","Data":"ece7daf002431675170cc62a2cbd7107d12d5b143b23081238adc818134a56d9"}
Oct 29 22:00:05.837965 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.837717    7082 scope.go:122] "RemoveContainer" containerID="ece7daf002431675170cc62a2cbd7107d12d5b143b23081238adc818134a56d9"
Oct 29 22:00:05.837965 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:05.837933    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888520-gfcjc_odh-ai-gateway-infra(b3e897f3-9840-42c8-8f21-b6ce623ef443)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc" podUID="b3e897f3-9840-42c8-8f21-b6ce623ef443"
Oct 29 22:00:05.840523 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.840482    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" event={"ID":"e9ac5793-4f48-4d6a-8390-19de6f99a814","Type":"ContainerStarted","Data":"dd2cb7245a324b8e286f9e9126733b31ef6ab17ef737554f47ade48e489de7c5"}
Oct 29 22:00:05.840523 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.840520    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" event={"ID":"e9ac5793-4f48-4d6a-8390-19de6f99a814","Type":"ContainerStarted","Data":"4b7ac3576c108eca9c6ce8c8ff368252ed2fca749f9694966f29272a6d9e112d"}
Oct 29 22:00:05.850137 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.850028    7082 scope.go:122] "RemoveContainer" containerID="df88969a3ddbbfd4a8c9b383b75f6fbb79bb75b3eb220d938a3f6425c418434d"
Oct 29 22:00:05.868150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.868121    7082 scope.go:122] "RemoveContainer" containerID="0bd8b70ac6d944545e265d4abf9e447e7dea7ee7914533080245c0df82bc4dbd"
Oct 29 22:00:05.908137 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.907769    7082 scope.go:122] "RemoveContainer" containerID="e7fed022052f37bfb1fa387f53a7f182ababe33b6b2ea19492bb28d8adc48e9a"
Oct 29 22:00:05.945730 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.945699    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:00:05.945865 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.945754    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 29 22:00:05.949863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.949836    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:05.978952 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:05.977829    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" podStartSLOduration=1.977809754 podStartE2EDuration="1.977809754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:05.963416878 +0000 UTC m=+778.463459177" watchObservedRunningTime="2026-10-29 22:00:05.977809754 +0000 UTC m=+778.477852054"
Oct 29 22:00:06.367130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.367098    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"]
Oct 29 22:00:06.383955 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.383906    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:06.384375 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.384348    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:06.384458 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.384403    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:06.549412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.547827    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:06.549412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.549248    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:00:06.549412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.549288    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:00:06.875498 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.875343    7082 scope.go:122] "RemoveContainer" containerID="6169044f24642d6c75cc04f7d31706c52e923e1bd4da907efaaf6a65328c3daf"
Oct 29 22:00:06.877946 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:06.877907    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8_odh-ai-gateway-infra(e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f"
Oct 29 22:00:06.880042 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.879760    7082 scope.go:122] "RemoveContainer" containerID="ece7daf002431675170cc62a2cbd7107d12d5b143b23081238adc818134a56d9"
Oct 29 22:00:06.880190 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:06.880113    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888520-gfcjc_odh-ai-gateway-infra(b3e897f3-9840-42c8-8f21-b6ce623ef443)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc" podUID="b3e897f3-9840-42c8-8f21-b6ce623ef443"
Oct 29 22:00:06.889654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.883928    7082 generic.go:356] "Generic (PLEG): container finished" podID="0fcd3c41-6238-4a00-af83-becbfaef0e0b" containerID="8e04e9429113bb3769e80dc4433ff815b0eb26282853a156937a50d6e628eebf" exitCode=60
Oct 29 22:00:06.889654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.884018    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv" event={"ID":"0fcd3c41-6238-4a00-af83-becbfaef0e0b","Type":"ContainerDied","Data":"8e04e9429113bb3769e80dc4433ff815b0eb26282853a156937a50d6e628eebf"}
Oct 29 22:00:06.889654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.884074    7082 scope.go:122] "RemoveContainer" containerID="68502db9d6b4788598589a6564543ad31ed3b130d3b699147af262112afbbc96"
Oct 29 22:00:06.889654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.884412    7082 scope.go:122] "RemoveContainer" containerID="8e04e9429113bb3769e80dc4433ff815b0eb26282853a156937a50d6e628eebf"
Oct 29 22:00:06.889654 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:06.884727    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv_odh-ai-gateway-infra(0fcd3c41-6238-4a00-af83-becbfaef0e0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv" podUID="0fcd3c41-6238-4a00-af83-becbfaef0e0b"
Oct 29 22:00:06.896721 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.896695    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" event={"ID":"694c4fbc-01cd-4141-8168-2a4c261e0103","Type":"ContainerStarted","Data":"75abdf7b32b7a08079db296a280754d8aca7410046713628c21a0189290f5d38"}
Oct 29 22:00:06.897270 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.896901    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" event={"ID":"694c4fbc-01cd-4141-8168-2a4c261e0103","Type":"ContainerStarted","Data":"2bf2e9a41218fa26d028a650c6baf6f69d7e31686c224136d046b1b7653865a0"}
Oct 29 22:00:06.912047 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.912022    7082 scope.go:122] "RemoveContainer" containerID="44c5fc77fafc81f1863acb8174578c91d9ae4bc2a019585047a81c772412fa1f"
Oct 29 22:00:06.912642 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:06.912590    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w_odh-ai-gateway-infra(051ea159-740b-4cc4-a5ef-1e9b03945cc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w" podUID="051ea159-740b-4cc4-a5ef-1e9b03945cc0"
Oct 29 22:00:06.915735 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.915713    7082 scope.go:122] "RemoveContainer" containerID="96bdb2cd8a7b18d5cd4df5f3c4865e0852ce1fb0fb36bb28cb99f240d27120df"
Oct 29 22:00:06.916141 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:06.916120    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh_odh-ai-gateway-infra(23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh" podUID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c"
Oct 29 22:00:06.946174 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.945815    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:00:06.946938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.946904    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 29 22:00:06.968022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:06.967994    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:07.220975 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:07.220857    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" podStartSLOduration=2.220834494 podStartE2EDuration="2.220834494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:07.157910619 +0000 UTC m=+779.657952910" watchObservedRunningTime="2026-10-29 22:00:07.220834494 +0000 UTC m=+779.720876794"
Oct 29 22:00:07.557354 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:07.557314    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:00:07.557674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:07.557618    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:00:07.959737 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:07.948069    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:00:07.959737 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:07.948126    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 29 22:00:07.969866 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:07.969231    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:00:07.969866 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:07.969277    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 29 22:00:07.975449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:07.975418    7082 scope.go:122] "RemoveContainer" containerID="8e04e9429113bb3769e80dc4433ff815b0eb26282853a156937a50d6e628eebf"
Oct 29 22:00:07.975818 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:07.975794    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv_odh-ai-gateway-infra(0fcd3c41-6238-4a00-af83-becbfaef0e0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv" podUID="0fcd3c41-6238-4a00-af83-becbfaef0e0b"
Oct 29 22:00:08.568654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:08.561418    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:00:08.568654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:08.561479    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:00:08.946150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:08.946052    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:00:08.946410 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:08.946376    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 29 22:00:08.951004 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:08.950978    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:00:08.951174 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:08.951151    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 29 22:00:09.550651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:09.550344    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:00:09.550651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:09.550408    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:00:09.950777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:09.950686    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:00:09.951043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:09.951014    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 29 22:00:09.951560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:09.951540    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Oct 29 22:00:09.951739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:09.951719    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Oct 29 22:00:10.553654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:10.553081    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:00:10.553654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:10.553140    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:00:10.971059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:10.970964    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:00:10.971319 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:10.971288    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 29 22:00:11.588806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:11.588767    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:00:11.618923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:11.599893    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:00:11.959195 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:11.959105    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:00:11.959439 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:11.959412    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 29 22:00:11.959619 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:11.959587    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:11.959765 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:11.959680    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:12.004761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.004731    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:12.008188 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.008161    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:12.020882 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.020854    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:00:12.548397 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.548365    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:00:12.548574 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.548412    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 29 22:00:12.818655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.818312    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"]
Oct 29 22:00:12.830078 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.830052    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"
Oct 29 22:00:12.838297 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.836923    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a65e410-qj4vg\" (UID: \"9031e15e-d7af-479a-8eb5-0b985f2fae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"
Oct 29 22:00:12.838297 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.837030    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a65e410-qj4vg\" (UID: \"9031e15e-d7af-479a-8eb5-0b985f2fae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"
Oct 29 22:00:12.838297 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.837092    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vblqp\" (UniqueName: \"kubernetes.io/projected/9031e15e-d7af-479a-8eb5-0b985f2fae99-kube-api-access-vblqp\") pod \"maas-api-revoke-sub-1a65e410-qj4vg\" (UID: \"9031e15e-d7af-479a-8eb5-0b985f2fae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"
Oct 29 22:00:12.842896 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.842303    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"]
Oct 29 22:00:12.846226 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.846183    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:00:12.948970 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.948932    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a65e410-qj4vg\" (UID: \"9031e15e-d7af-479a-8eb5-0b985f2fae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"
Oct 29 22:00:12.978167 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.978123    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a65e410-qj4vg\" (UID: \"9031e15e-d7af-479a-8eb5-0b985f2fae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"
Oct 29 22:00:12.978523 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.978492    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vblqp\" (UniqueName: \"kubernetes.io/projected/9031e15e-d7af-479a-8eb5-0b985f2fae99-kube-api-access-vblqp\") pod \"maas-api-revoke-sub-1a65e410-qj4vg\" (UID: \"9031e15e-d7af-479a-8eb5-0b985f2fae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"
Oct 29 22:00:12.979287 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.949986    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a65e410-qj4vg\" (UID: \"9031e15e-d7af-479a-8eb5-0b985f2fae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"
Oct 29 22:00:12.980459 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.980431    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:00:12.980756 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.980711    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 29 22:00:12.997917 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:12.997871    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a65e410-qj4vg\" (UID: \"9031e15e-d7af-479a-8eb5-0b985f2fae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"
Oct 29 22:00:13.157700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:13.156608    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vblqp\" (UniqueName: \"kubernetes.io/projected/9031e15e-d7af-479a-8eb5-0b985f2fae99-kube-api-access-vblqp\") pod \"maas-api-revoke-sub-1a65e410-qj4vg\" (UID: \"9031e15e-d7af-479a-8eb5-0b985f2fae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"
Oct 29 22:00:13.453066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:13.452803    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"
Oct 29 22:00:13.808954 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:13.808915    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:13.809987 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:13.809963    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:13.811540 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:13.811521    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:00:13.850256 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:00:13.850227    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9031e15e_d7af_479a_8eb5_0b985f2fae99.slice/crio-43c7c36256d23638b9398764011cf781d490e13e7f6a81d1d7cc9aa41b0a462f WatchSource:0}: Error finding container 43c7c36256d23638b9398764011cf781d490e13e7f6a81d1d7cc9aa41b0a462f: Status 404 returned error can't find the container with id 43c7c36256d23638b9398764011cf781d490e13e7f6a81d1d7cc9aa41b0a462f
Oct 29 22:00:13.863791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:13.863762    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"]
Oct 29 22:00:13.975984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:13.955013    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:00:13.975984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:13.975797    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 29 22:00:14.325923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:14.325891    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg" event={"ID":"9031e15e-d7af-479a-8eb5-0b985f2fae99","Type":"ContainerStarted","Data":"43c7c36256d23638b9398764011cf781d490e13e7f6a81d1d7cc9aa41b0a462f"}
Oct 29 22:00:14.955078 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:14.955049    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:14.955471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:14.955408    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:14.956158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:14.956136    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:00:15.337059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:15.337027    7082 generic.go:356] "Generic (PLEG): container finished" podID="9031e15e-d7af-479a-8eb5-0b985f2fae99" containerID="a5b141e06198f90c744b0ae020c887f79fbeb2c039269f8823b1cf0941e00d30" exitCode=0
Oct 29 22:00:15.337250 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:15.337101    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg" event={"ID":"9031e15e-d7af-479a-8eb5-0b985f2fae99","Type":"ContainerDied","Data":"a5b141e06198f90c744b0ae020c887f79fbeb2c039269f8823b1cf0941e00d30"}
Oct 29 22:00:15.624398 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:15.624312    7082 scope.go:122] "RemoveContainer" containerID="44c5fc77fafc81f1863acb8174578c91d9ae4bc2a019585047a81c772412fa1f"
Oct 29 22:00:16.347338 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:16.347301    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w" event={"ID":"051ea159-740b-4cc4-a5ef-1e9b03945cc0","Type":"ContainerStarted","Data":"96503df615af082fe70c42e7614991b02b9f20ade7ae293c4da6cad7843e2dcf"}
Oct 29 22:00:16.526786 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:16.526747    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:00:16.545187 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:16.545151    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"
Oct 29 22:00:16.625058 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:16.624981    7082 scope.go:122] "RemoveContainer" containerID="ece7daf002431675170cc62a2cbd7107d12d5b143b23081238adc818134a56d9"
Oct 29 22:00:16.642463 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:16.642411    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-openshift-ca\") pod \"9031e15e-d7af-479a-8eb5-0b985f2fae99\" (UID: \"9031e15e-d7af-479a-8eb5-0b985f2fae99\") "
Oct 29 22:00:16.642604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:16.642559    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-xks-ca\") pod \"9031e15e-d7af-479a-8eb5-0b985f2fae99\" (UID: \"9031e15e-d7af-479a-8eb5-0b985f2fae99\") "
Oct 29 22:00:16.642712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:16.642611    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9031e15e-d7af-479a-8eb5-0b985f2fae99-kube-api-access-vblqp\" (UniqueName: \"kubernetes.io/projected/9031e15e-d7af-479a-8eb5-0b985f2fae99-kube-api-access-vblqp\") pod \"9031e15e-d7af-479a-8eb5-0b985f2fae99\" (UID: \"9031e15e-d7af-479a-8eb5-0b985f2fae99\") "
Oct 29 22:00:16.648206 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:16.648090    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9031e15e-d7af-479a-8eb5-0b985f2fae99-kube-api-access-vblqp" pod "9031e15e-d7af-479a-8eb5-0b985f2fae99" (UID: "9031e15e-d7af-479a-8eb5-0b985f2fae99"). InnerVolumeSpecName "kube-api-access-vblqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:16.649553 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:16.648784    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-openshift-ca" pod "9031e15e-d7af-479a-8eb5-0b985f2fae99" (UID: "9031e15e-d7af-479a-8eb5-0b985f2fae99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:00:16.658170 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:16.658141    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-xks-ca" pod "9031e15e-d7af-479a-8eb5-0b985f2fae99" (UID: "9031e15e-d7af-479a-8eb5-0b985f2fae99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:00:16.744693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:16.744654    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:16.744693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:16.744693    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9031e15e-d7af-479a-8eb5-0b985f2fae99-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:16.744897 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:16.744709    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vblqp\" (UniqueName: \"kubernetes.io/projected/9031e15e-d7af-479a-8eb5-0b985f2fae99-kube-api-access-vblqp\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:17.438675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.437766    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"]
Oct 29 22:00:17.450278 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.449673    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc" event={"ID":"b3e897f3-9840-42c8-8f21-b6ce623ef443","Type":"ContainerStarted","Data":"6fdf695a12a77b04b611864289faee05d284b0f8f44915338114e629c33ecf85"}
Oct 29 22:00:17.454660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.454603    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg" event={"ID":"9031e15e-d7af-479a-8eb5-0b985f2fae99","Type":"ContainerDied","Data":"43c7c36256d23638b9398764011cf781d490e13e7f6a81d1d7cc9aa41b0a462f"}
Oct 29 22:00:17.454923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.454906    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43c7c36256d23638b9398764011cf781d490e13e7f6a81d1d7cc9aa41b0a462f"
Oct 29 22:00:17.455122 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.455109    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"
Oct 29 22:00:17.466688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.466653    7082 generic.go:356] "Generic (PLEG): container finished" podID="051ea159-740b-4cc4-a5ef-1e9b03945cc0" containerID="96503df615af082fe70c42e7614991b02b9f20ade7ae293c4da6cad7843e2dcf" exitCode=60
Oct 29 22:00:17.467155 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.467128    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w" event={"ID":"051ea159-740b-4cc4-a5ef-1e9b03945cc0","Type":"ContainerDied","Data":"96503df615af082fe70c42e7614991b02b9f20ade7ae293c4da6cad7843e2dcf"}
Oct 29 22:00:17.467497 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.467479    7082 scope.go:122] "RemoveContainer" containerID="44c5fc77fafc81f1863acb8174578c91d9ae4bc2a019585047a81c772412fa1f"
Oct 29 22:00:17.700474 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.700408    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"
Oct 29 22:00:17.779058 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.779029    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051ea159-740b-4cc4-a5ef-1e9b03945cc0-kube-api-access-59ddz\" (UniqueName: \"kubernetes.io/projected/051ea159-740b-4cc4-a5ef-1e9b03945cc0-kube-api-access-59ddz\") pod \"051ea159-740b-4cc4-a5ef-1e9b03945cc0\" (UID: \"051ea159-740b-4cc4-a5ef-1e9b03945cc0\") "
Oct 29 22:00:17.779257 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.779126    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051ea159-740b-4cc4-a5ef-1e9b03945cc0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/051ea159-740b-4cc4-a5ef-1e9b03945cc0-maas-api-service-ca\") pod \"051ea159-740b-4cc4-a5ef-1e9b03945cc0\" (UID: \"051ea159-740b-4cc4-a5ef-1e9b03945cc0\") "
Oct 29 22:00:17.779539 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.779516    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051ea159-740b-4cc4-a5ef-1e9b03945cc0-maas-api-service-ca" pod "051ea159-740b-4cc4-a5ef-1e9b03945cc0" (UID: "051ea159-740b-4cc4-a5ef-1e9b03945cc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:00:17.780238 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.780213    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/051ea159-740b-4cc4-a5ef-1e9b03945cc0-maas-api-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:17.808072 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.788190    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051ea159-740b-4cc4-a5ef-1e9b03945cc0-kube-api-access-59ddz" pod "051ea159-740b-4cc4-a5ef-1e9b03945cc0" (UID: "051ea159-740b-4cc4-a5ef-1e9b03945cc0"). InnerVolumeSpecName "kube-api-access-59ddz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:17.887186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:17.886706    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59ddz\" (UniqueName: \"kubernetes.io/projected/051ea159-740b-4cc4-a5ef-1e9b03945cc0-kube-api-access-59ddz\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:18.483888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:18.483330    7082 generic.go:356] "Generic (PLEG): container finished" podID="b3e897f3-9840-42c8-8f21-b6ce623ef443" containerID="6fdf695a12a77b04b611864289faee05d284b0f8f44915338114e629c33ecf85" exitCode=60
Oct 29 22:00:18.483888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:18.483503    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc" event={"ID":"b3e897f3-9840-42c8-8f21-b6ce623ef443","Type":"ContainerDied","Data":"6fdf695a12a77b04b611864289faee05d284b0f8f44915338114e629c33ecf85"}
Oct 29 22:00:18.483888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:18.483556    7082 scope.go:122] "RemoveContainer" containerID="ece7daf002431675170cc62a2cbd7107d12d5b143b23081238adc818134a56d9"
Oct 29 22:00:18.484501 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:18.484024    7082 scope.go:122] "RemoveContainer" containerID="6fdf695a12a77b04b611864289faee05d284b0f8f44915338114e629c33ecf85"
Oct 29 22:00:18.484501 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:18.484377    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888520-gfcjc_odh-ai-gateway-infra(b3e897f3-9840-42c8-8f21-b6ce623ef443)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc" podUID="b3e897f3-9840-42c8-8f21-b6ce623ef443"
Oct 29 22:00:18.502922 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:18.491731    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w" event={"ID":"051ea159-740b-4cc4-a5ef-1e9b03945cc0","Type":"ContainerDied","Data":"5dae207ea4975838880f9636f7a54c701813d38a51f1b68745b2d98fd2149ba2"}
Oct 29 22:00:18.502922 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:18.491840    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"
Oct 29 22:00:18.587714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:18.561834    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"]
Oct 29 22:00:18.634979 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:18.627371    7082 scope.go:122] "RemoveContainer" containerID="96bdb2cd8a7b18d5cd4df5f3c4865e0852ce1fb0fb36bb28cb99f240d27120df"
Oct 29 22:00:18.651653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:18.643814    7082 scope.go:122] "RemoveContainer" containerID="8e04e9429113bb3769e80dc4433ff815b0eb26282853a156937a50d6e628eebf"
Oct 29 22:00:18.747308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:18.692641    7082 scope.go:122] "RemoveContainer" containerID="96503df615af082fe70c42e7614991b02b9f20ade7ae293c4da6cad7843e2dcf"
Oct 29 22:00:18.904649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:18.903697    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"]
Oct 29 22:00:18.917648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:18.910559    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888520-lpx7w"]
Oct 29 22:00:19.753191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:19.753106    7082 scope.go:122] "RemoveContainer" containerID="6169044f24642d6c75cc04f7d31706c52e923e1bd4da907efaaf6a65328c3daf"
Oct 29 22:00:20.063617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:20.059252    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051ea159-740b-4cc4-a5ef-1e9b03945cc0" path="/var/lib/kubelet/pods/051ea159-740b-4cc4-a5ef-1e9b03945cc0/volumes"
Oct 29 22:00:20.085851 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:20.085817    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv" event={"ID":"0fcd3c41-6238-4a00-af83-becbfaef0e0b","Type":"ContainerStarted","Data":"05fc085ea97dacbf653c3766fef4cfd01d88d4c3a9cf89a4ff8f717c4643bcc2"}
Oct 29 22:00:20.118303 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:20.113143    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:20.169879 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:20.152127    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh" event={"ID":"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c","Type":"ContainerStarted","Data":"3ef9529cc2ba41ce495d49c5208d0441cfe56c36eaf8a50b57fd95dccd5eba99"}
Oct 29 22:00:20.274673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:20.273948    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:20.708576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:20.705739    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:00:20.958136 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:20.935041    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.150531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:21.150453    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.376138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:21.254593    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.415680 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:21.338152    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.415680 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:21.400092    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:00:21.415680 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:21.400205    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:00:21.425988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:21.425949    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.760341 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:21.728410    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.938303 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:21.938266    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.011919 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.011687    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:00:22.012511 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.012493    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:00:22.012700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.012660    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"]
Oct 29 22:00:22.201652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.168769    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.376651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.375822    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.376651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.376134    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:00:22.381654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.376932    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:00:22.450768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.450724    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:00:22.461119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.451704    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.461552 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.461534    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:00:22.472716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.472683    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"]
Oct 29 22:00:22.602737 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.581234    7082 generic.go:356] "Generic (PLEG): container finished" podID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" containerID="3ef9529cc2ba41ce495d49c5208d0441cfe56c36eaf8a50b57fd95dccd5eba99" exitCode=60
Oct 29 22:00:22.602737 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.581389    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh" event={"ID":"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c","Type":"ContainerDied","Data":"3ef9529cc2ba41ce495d49c5208d0441cfe56c36eaf8a50b57fd95dccd5eba99"}
Oct 29 22:00:22.602737 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.581470    7082 scope.go:122] "RemoveContainer" containerID="96bdb2cd8a7b18d5cd4df5f3c4865e0852ce1fb0fb36bb28cb99f240d27120df"
Oct 29 22:00:22.662226 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.662193    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"
Oct 29 22:00:22.739654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.720200    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8" event={"ID":"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f","Type":"ContainerStarted","Data":"ffcc1b1baf284ff948ab729eb2d2a280e7089ba50087a787794675c0c2ad8819"}
Oct 29 22:00:22.778645 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.748292    7082 scope.go:122] "RemoveContainer" containerID="ffcc1b1baf284ff948ab729eb2d2a280e7089ba50087a787794675c0c2ad8819"
Oct 29 22:00:22.780113 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:22.780075    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8_odh-ai-gateway-infra(e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f"
Oct 29 22:00:22.897750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.795481    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3e897f3-9840-42c8-8f21-b6ce623ef443-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e897f3-9840-42c8-8f21-b6ce623ef443-maas-api-service-ca\") pod \"b3e897f3-9840-42c8-8f21-b6ce623ef443\" (UID: \"b3e897f3-9840-42c8-8f21-b6ce623ef443\") "
Oct 29 22:00:22.922844 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.922805    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3e897f3-9840-42c8-8f21-b6ce623ef443-kube-api-access-sg6vw\" (UniqueName: \"kubernetes.io/projected/b3e897f3-9840-42c8-8f21-b6ce623ef443-kube-api-access-sg6vw\") pod \"b3e897f3-9840-42c8-8f21-b6ce623ef443\" (UID: \"b3e897f3-9840-42c8-8f21-b6ce623ef443\") "
Oct 29 22:00:22.940890 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.940851    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e897f3-9840-42c8-8f21-b6ce623ef443-maas-api-service-ca" pod "b3e897f3-9840-42c8-8f21-b6ce623ef443" (UID: "b3e897f3-9840-42c8-8f21-b6ce623ef443"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:00:23.089946 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.986307    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:23.089946 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:22.987373    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:23.089946 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:23.027734    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3e897f3-9840-42c8-8f21-b6ce623ef443-maas-api-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:23.089946 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:23.045751    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:23.089946 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:23.045814    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" podUID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:23.502708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:23.494819    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e897f3-9840-42c8-8f21-b6ce623ef443-kube-api-access-sg6vw" pod "b3e897f3-9840-42c8-8f21-b6ce623ef443" (UID: "b3e897f3-9840-42c8-8f21-b6ce623ef443"). InnerVolumeSpecName "kube-api-access-sg6vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:23.563654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:23.523307    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:00:23.958277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:23.529801    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:00:23.958277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:23.530698    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:23.958277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:23.530730    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:00:23.958277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:23.531363    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg6vw\" (UniqueName: \"kubernetes.io/projected/b3e897f3-9840-42c8-8f21-b6ce623ef443-kube-api-access-sg6vw\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:23.958277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:23.626183    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:23.958277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:23.737334    7082 generic.go:356] "Generic (PLEG): container finished" podID="0fcd3c41-6238-4a00-af83-becbfaef0e0b" containerID="05fc085ea97dacbf653c3766fef4cfd01d88d4c3a9cf89a4ff8f717c4643bcc2" exitCode=60
Oct 29 22:00:24.000362 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:23.992338    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:00:24.000362 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:23.993069    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:00:24.979128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:24.979062    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:00:25.005049 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:25.004998    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.072681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:25.023619    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:00:25.197067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:25.093262    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:25.197067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:25.196828    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.197067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:25.195295    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.258759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:25.252136    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.258759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:25.252767    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:00:25.524717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:25.375733    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:00:25.524717 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:25.453870    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.827s"
Oct 29 22:00:25.524717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:25.453908    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv" event={"ID":"0fcd3c41-6238-4a00-af83-becbfaef0e0b","Type":"ContainerDied","Data":"05fc085ea97dacbf653c3766fef4cfd01d88d4c3a9cf89a4ff8f717c4643bcc2"}
Oct 29 22:00:25.524717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:25.454000    7082 scope.go:122] "RemoveContainer" containerID="8e04e9429113bb3769e80dc4433ff815b0eb26282853a156937a50d6e628eebf"
Oct 29 22:00:25.670865 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:25.670835    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"
Oct 29 22:00:25.972770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:25.969889    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-maas-api-service-ca\") pod \"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c\" (UID: \"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c\") "
Oct 29 22:00:26.268089 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.267990    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-kube-api-access-q6prp\" (UniqueName: \"kubernetes.io/projected/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-kube-api-access-q6prp\") pod \"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c\" (UID: \"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c\") "
Oct 29 22:00:26.297868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.013758    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.311214 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.014038    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.330804 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.206620    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.330804 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:25.981482    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-maas-api-service-ca" pod "23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" (UID: "23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:00:26.394960 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.381291    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:00:26.394960 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.381300    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.394960 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.383788    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.413655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.406002    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-maas-api-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:26.433449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.428281    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:26.433449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.428361    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.456842 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.456800    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-kube-api-access-q6prp" pod "23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" (UID: "23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c"). InnerVolumeSpecName "kube-api-access-q6prp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:26.462649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.460936    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"
Oct 29 22:00:26.462649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.461169    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc" event={"ID":"b3e897f3-9840-42c8-8f21-b6ce623ef443","Type":"ContainerDied","Data":"f01cda640bb71991ed03a8635d0e84bebb6be59715c015e503f6e87d47acfcf5"}
Oct 29 22:00:26.462649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.461247    7082 scope.go:122] "RemoveContainer" containerID="6fdf695a12a77b04b611864289faee05d284b0f8f44915338114e629c33ecf85"
Oct 29 22:00:26.485963 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.485937    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh" event={"ID":"23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c","Type":"ContainerDied","Data":"795d185662eaffb31f6ea16216dcab02ed93202372e8f56f9bc20288b58ad0f1"}
Oct 29 22:00:26.486261 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.486235    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"
Oct 29 22:00:26.552777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.540452    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6prp\" (UniqueName: \"kubernetes.io/projected/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c-kube-api-access-q6prp\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:26.601975 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.601933    7082 generic.go:356] "Generic (PLEG): container finished" podID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" containerID="ffcc1b1baf284ff948ab729eb2d2a280e7089ba50087a787794675c0c2ad8819" exitCode=60
Oct 29 22:00:26.602111 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.602084    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8" event={"ID":"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f","Type":"ContainerDied","Data":"ffcc1b1baf284ff948ab729eb2d2a280e7089ba50087a787794675c0c2ad8819"}
Oct 29 22:00:26.604667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.602598    7082 scope.go:122] "RemoveContainer" containerID="ffcc1b1baf284ff948ab729eb2d2a280e7089ba50087a787794675c0c2ad8819"
Oct 29 22:00:26.604667 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:26.603101    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8_odh-ai-gateway-infra(e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.639569    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"]
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640745    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9031e15e-d7af-479a-8eb5-0b985f2fae99" containerName="revoke-keys"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640767    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9031e15e-d7af-479a-8eb5-0b985f2fae99" containerName="revoke-keys"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640810    7082 cpu_manager.go:414] "containerMap: removing container" podUID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640818    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640843    7082 cpu_manager.go:414] "containerMap: removing container" podUID="051ea159-740b-4cc4-a5ef-1e9b03945cc0" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640850    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051ea159-740b-4cc4-a5ef-1e9b03945cc0" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640866    7082 cpu_manager.go:414] "containerMap: removing container" podUID="b3e897f3-9840-42c8-8f21-b6ce623ef443" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640887    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e897f3-9840-42c8-8f21-b6ce623ef443" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640900    7082 cpu_manager.go:414] "containerMap: removing container" podUID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640907    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640936    7082 cpu_manager.go:414] "containerMap: removing container" podUID="b3e897f3-9840-42c8-8f21-b6ce623ef443" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640943    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e897f3-9840-42c8-8f21-b6ce623ef443" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640971    7082 cpu_manager.go:414] "containerMap: removing container" podUID="051ea159-740b-4cc4-a5ef-1e9b03945cc0" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640978    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051ea159-740b-4cc4-a5ef-1e9b03945cc0" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640988    7082 cpu_manager.go:414] "containerMap: removing container" podUID="b3e897f3-9840-42c8-8f21-b6ce623ef443" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.640995    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e897f3-9840-42c8-8f21-b6ce623ef443" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.641136    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.641150    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="051ea159-740b-4cc4-a5ef-1e9b03945cc0" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.641160    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e897f3-9840-42c8-8f21-b6ce623ef443" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.641169    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.641180    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e897f3-9840-42c8-8f21-b6ce623ef443" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.641193    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="051ea159-740b-4cc4-a5ef-1e9b03945cc0" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.641201    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e897f3-9840-42c8-8f21-b6ce623ef443" containerName="cleanup"
Oct 29 22:00:26.642868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.641212    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="9031e15e-d7af-479a-8eb5-0b985f2fae99" containerName="revoke-keys"
Oct 29 22:00:26.737500 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.723349    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:26.737500 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.723408    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.815901 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.805488    7082 scope.go:122] "RemoveContainer" containerID="3ef9529cc2ba41ce495d49c5208d0441cfe56c36eaf8a50b57fd95dccd5eba99"
Oct 29 22:00:26.857692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.853690    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"]
Oct 29 22:00:26.857692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.854555    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"
Oct 29 22:00:26.896182 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.896073    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:00:26.977822 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.977784    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0563756-7482d\" (UID: \"02f42909-c84a-49fa-97aa-d65fb90ff51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"
Oct 29 22:00:26.978085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.978065    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0563756-7482d\" (UID: \"02f42909-c84a-49fa-97aa-d65fb90ff51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"
Oct 29 22:00:26.981258 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:26.978268    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw4cm\" (UniqueName: \"kubernetes.io/projected/02f42909-c84a-49fa-97aa-d65fb90ff51b-kube-api-access-dw4cm\") pod \"maas-api-revoke-sub-e0563756-7482d\" (UID: \"02f42909-c84a-49fa-97aa-d65fb90ff51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"
Oct 29 22:00:27.096707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.086356    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0563756-7482d\" (UID: \"02f42909-c84a-49fa-97aa-d65fb90ff51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"
Oct 29 22:00:27.096707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.086419    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0563756-7482d\" (UID: \"02f42909-c84a-49fa-97aa-d65fb90ff51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"
Oct 29 22:00:27.096707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.086501    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw4cm\" (UniqueName: \"kubernetes.io/projected/02f42909-c84a-49fa-97aa-d65fb90ff51b-kube-api-access-dw4cm\") pod \"maas-api-revoke-sub-e0563756-7482d\" (UID: \"02f42909-c84a-49fa-97aa-d65fb90ff51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"
Oct 29 22:00:27.096707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.087184    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0563756-7482d\" (UID: \"02f42909-c84a-49fa-97aa-d65fb90ff51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"
Oct 29 22:00:27.300655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.252403    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw4cm\" (UniqueName: \"kubernetes.io/projected/02f42909-c84a-49fa-97aa-d65fb90ff51b-kube-api-access-dw4cm\") pod \"maas-api-revoke-sub-e0563756-7482d\" (UID: \"02f42909-c84a-49fa-97aa-d65fb90ff51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"
Oct 29 22:00:27.344781 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.299261    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:00:27.372707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.316900    7082 scope.go:122] "RemoveContainer" containerID="6169044f24642d6c75cc04f7d31706c52e923e1bd4da907efaaf6a65328c3daf"
Oct 29 22:00:27.373019 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.317167    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:27.394656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.317206    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:00:27.478195 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.345947    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"]
Oct 29 22:00:27.478195 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.388227    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:00:27.478195 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.388243    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:00:27.478195 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.388274    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:27.478195 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.389063    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0563756-7482d\" (UID: \"02f42909-c84a-49fa-97aa-d65fb90ff51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"
Oct 29 22:00:27.584658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.583909    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"
Oct 29 22:00:27.940787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:27.940736    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:28.187250 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.177299    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:28.187250 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.177474    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:28.356654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.348449    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:28.542710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.377343    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"]
Oct 29 22:00:28.546753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.546721    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888520-gfcjc"]
Oct 29 22:00:28.550722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.389806    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:28.558993 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.558957    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:28.587011 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.389733    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:28.587531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.572128    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:28.587929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.573241    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:28.606750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.574233    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:00:28.627473 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.627421    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:00:28.771114 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.770974    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:28.771559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:28.771528    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:29.557714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:29.550152    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded"
Oct 29 22:00:29.557714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:29.550529    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:29.608651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:29.607850    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:00:29.627727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:29.627680    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:00:29.639250 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:29.639199    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:29.640223 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:29.640177    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:00:29.699186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:29.699030    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:29.878927 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:29.878862    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:30.571422 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.571376    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:30.628920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.628853    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:00:30.642569 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.641868    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:00:30.670654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.670228    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" podUID="4684c6fd-8ac7-4005-a20d-48396c0ab981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:30.675057 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.675014    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3e897f3-9840-42c8-8f21-b6ce623ef443" path="/var/lib/kubelet/pods/b3e897f3-9840-42c8-8f21-b6ce623ef443/volumes"
Oct 29 22:00:30.679353 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:30.675562    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.034s"
Oct 29 22:00:30.679353 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.675734    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:00:30.679353 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.675788    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:00:30.679353 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.675859    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"]
Oct 29 22:00:30.679353 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.675888    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a3c02e-29888520-npmbh"]
Oct 29 22:00:30.747983 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:30.717369    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:00:30.748277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.748000    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:30.754027 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.749552    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:00:30.763768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.758819    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"
Oct 29 22:00:30.962130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.962093    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv" event={"ID":"0fcd3c41-6238-4a00-af83-becbfaef0e0b","Type":"ContainerDied","Data":"3e8c207a57d9db020b0d16a5aa210cd762d61dc0c224ff4dd5155f55d3d5a651"}
Oct 29 22:00:30.962383 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:30.962363    7082 scope.go:122] "RemoveContainer" containerID="05fc085ea97dacbf653c3766fef4cfd01d88d4c3a9cf89a4ff8f717c4643bcc2"
Oct 29 22:00:31.000503 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.000042    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:00:31.031737 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.031710    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fcd3c41-6238-4a00-af83-becbfaef0e0b-kube-api-access-cjv8n\" (UniqueName: \"kubernetes.io/projected/0fcd3c41-6238-4a00-af83-becbfaef0e0b-kube-api-access-cjv8n\") pod \"0fcd3c41-6238-4a00-af83-becbfaef0e0b\" (UID: \"0fcd3c41-6238-4a00-af83-becbfaef0e0b\") "
Oct 29 22:00:31.032046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.032028    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fcd3c41-6238-4a00-af83-becbfaef0e0b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd3c41-6238-4a00-af83-becbfaef0e0b-maas-api-service-ca\") pod \"0fcd3c41-6238-4a00-af83-becbfaef0e0b\" (UID: \"0fcd3c41-6238-4a00-af83-becbfaef0e0b\") "
Oct 29 22:00:31.038143 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.038080    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:00:31.054974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.051442    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fcd3c41-6238-4a00-af83-becbfaef0e0b-maas-api-service-ca" pod "0fcd3c41-6238-4a00-af83-becbfaef0e0b" (UID: "0fcd3c41-6238-4a00-af83-becbfaef0e0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:00:31.054974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.053141    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:00:31.054974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.054142    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"
Oct 29 22:00:31.080489 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.080460    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fcd3c41-6238-4a00-af83-becbfaef0e0b-kube-api-access-cjv8n" pod "0fcd3c41-6238-4a00-af83-becbfaef0e0b" (UID: "0fcd3c41-6238-4a00-af83-becbfaef0e0b"). InnerVolumeSpecName "kube-api-access-cjv8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:31.134789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.134337    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-kube-api-access-qb8hw\" (UniqueName: \"kubernetes.io/projected/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-kube-api-access-qb8hw\") pod \"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f\" (UID: \"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f\") "
Oct 29 22:00:31.135154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.135131    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-maas-api-service-ca\") pod \"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f\" (UID: \"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f\") "
Oct 29 22:00:31.136779 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.135478    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-maas-api-service-ca" pod "e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" (UID: "e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:00:31.137939 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.137883    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjv8n\" (UniqueName: \"kubernetes.io/projected/0fcd3c41-6238-4a00-af83-becbfaef0e0b-kube-api-access-cjv8n\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:31.137939 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.137910    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-maas-api-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:31.137939 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.137926    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd3c41-6238-4a00-af83-becbfaef0e0b-maas-api-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:31.140260 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.140236    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-kube-api-access-qb8hw" pod "e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" (UID: "e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f"). InnerVolumeSpecName "kube-api-access-qb8hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:31.200970 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:00:31.200930    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f42909_c84a_49fa_97aa_d65fb90ff51b.slice/crio-e8b53867914cf1a5cffb09d40c81c90d651af960e40b1e7e20b7399d5e8ec022 WatchSource:0}: Error finding container e8b53867914cf1a5cffb09d40c81c90d651af960e40b1e7e20b7399d5e8ec022: Status 404 returned error can't find the container with id e8b53867914cf1a5cffb09d40c81c90d651af960e40b1e7e20b7399d5e8ec022
Oct 29 22:00:31.242245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.240685    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"]
Oct 29 22:00:31.245225 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.244831    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb8hw\" (UniqueName: \"kubernetes.io/projected/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f-kube-api-access-qb8hw\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:31.634702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:31.634667    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" path="/var/lib/kubelet/pods/23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c/volumes"
Oct 29 22:00:32.058986 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:32.058947    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"
Oct 29 22:00:32.059180 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:32.059007    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8" event={"ID":"e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f","Type":"ContainerDied","Data":"9d8b66c7ac9a4448ece1a03603670f8f938cc39565c27f4ecb299c47b9c8a1ab"}
Oct 29 22:00:32.059180 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:32.059071    7082 scope.go:122] "RemoveContainer" containerID="ffcc1b1baf284ff948ab729eb2d2a280e7089ba50087a787794675c0c2ad8819"
Oct 29 22:00:32.060460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:32.060432    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"
Oct 29 22:00:32.062727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:32.062697    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d" event={"ID":"02f42909-c84a-49fa-97aa-d65fb90ff51b","Type":"ContainerStarted","Data":"3648544a56d4fc16c3f5f186cd49f45e5ee21907d9c21eedac978a1f034465e1"}
Oct 29 22:00:32.062833 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:32.062747    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d" event={"ID":"02f42909-c84a-49fa-97aa-d65fb90ff51b","Type":"ContainerStarted","Data":"e8b53867914cf1a5cffb09d40c81c90d651af960e40b1e7e20b7399d5e8ec022"}
Oct 29 22:00:32.116302 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:32.116265    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"]
Oct 29 22:00:32.137808 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:32.137773    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a9def7-29888520-8s9vv"]
Oct 29 22:00:32.174278 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:32.174229    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d" podStartSLOduration=10.174214966 podStartE2EDuration="10.174214966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:32.170278178 +0000 UTC m=+804.670320478" watchObservedRunningTime="2026-10-29 22:00:32.174214966 +0000 UTC m=+804.674257278"
Oct 29 22:00:32.210869 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:32.210839    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"]
Oct 29 22:00:32.228925 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:32.226886    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b31f3f-29888520-c8gw8"]
Oct 29 22:00:33.071707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.071673    7082 generic.go:356] "Generic (PLEG): container finished" podID="02f42909-c84a-49fa-97aa-d65fb90ff51b" containerID="3648544a56d4fc16c3f5f186cd49f45e5ee21907d9c21eedac978a1f034465e1" exitCode=0
Oct 29 22:00:33.072146 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.071734    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d" event={"ID":"02f42909-c84a-49fa-97aa-d65fb90ff51b","Type":"ContainerDied","Data":"3648544a56d4fc16c3f5f186cd49f45e5ee21907d9c21eedac978a1f034465e1"}
Oct 29 22:00:33.240212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240180    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"]
Oct 29 22:00:33.240880 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240867    7082 cpu_manager.go:414] "containerMap: removing container" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" containerName="cleanup"
Oct 29 22:00:33.240934 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240882    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" containerName="cleanup"
Oct 29 22:00:33.240934 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240897    7082 cpu_manager.go:414] "containerMap: removing container" podUID="0fcd3c41-6238-4a00-af83-becbfaef0e0b" containerName="cleanup"
Oct 29 22:00:33.240934 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240902    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fcd3c41-6238-4a00-af83-becbfaef0e0b" containerName="cleanup"
Oct 29 22:00:33.240934 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240926    7082 cpu_manager.go:414] "containerMap: removing container" podUID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" containerName="cleanup"
Oct 29 22:00:33.240934 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240931    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" containerName="cleanup"
Oct 29 22:00:33.241085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240941    7082 cpu_manager.go:414] "containerMap: removing container" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" containerName="cleanup"
Oct 29 22:00:33.241085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240946    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" containerName="cleanup"
Oct 29 22:00:33.241085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240955    7082 cpu_manager.go:414] "containerMap: removing container" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" containerName="cleanup"
Oct 29 22:00:33.241085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240960    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" containerName="cleanup"
Oct 29 22:00:33.241085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240977    7082 cpu_manager.go:414] "containerMap: removing container" podUID="0fcd3c41-6238-4a00-af83-becbfaef0e0b" containerName="cleanup"
Oct 29 22:00:33.241085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240981    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fcd3c41-6238-4a00-af83-becbfaef0e0b" containerName="cleanup"
Oct 29 22:00:33.241085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240987    7082 cpu_manager.go:414] "containerMap: removing container" podUID="051ea159-740b-4cc4-a5ef-1e9b03945cc0" containerName="cleanup"
Oct 29 22:00:33.241085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.240993    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051ea159-740b-4cc4-a5ef-1e9b03945cc0" containerName="cleanup"
Oct 29 22:00:33.241085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.241080    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fcd3c41-6238-4a00-af83-becbfaef0e0b" containerName="cleanup"
Oct 29 22:00:33.241085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.241087    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="051ea159-740b-4cc4-a5ef-1e9b03945cc0" containerName="cleanup"
Oct 29 22:00:33.241352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.241095    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" containerName="cleanup"
Oct 29 22:00:33.241352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.241101    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fcd3c41-6238-4a00-af83-becbfaef0e0b" containerName="cleanup"
Oct 29 22:00:33.241352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.241109    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e43b1f-b6b3-41b1-9a51-ddd9b88cfc0c" containerName="cleanup"
Oct 29 22:00:33.241352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.241114    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" containerName="cleanup"
Oct 29 22:00:33.244143 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.244126    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"
Oct 29 22:00:33.266329 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.266302    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"]
Oct 29 22:00:33.379058 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.378980    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"]
Oct 29 22:00:33.379809 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.379787    7082 cpu_manager.go:414] "containerMap: removing container" podUID="0fcd3c41-6238-4a00-af83-becbfaef0e0b" containerName="cleanup"
Oct 29 22:00:33.379809 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.379806    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fcd3c41-6238-4a00-af83-becbfaef0e0b" containerName="cleanup"
Oct 29 22:00:33.379989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.379920    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fcd3c41-6238-4a00-af83-becbfaef0e0b" containerName="cleanup"
Oct 29 22:00:33.379989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.379934    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" containerName="cleanup"
Oct 29 22:00:33.383549 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.383528    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.391322 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.391297    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45de20f3-8ljw9\" (UID: \"43ded71e-f4fa-4e66-a501-6f8792598ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"
Oct 29 22:00:33.392926 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.392902    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q69fx\" (UniqueName: \"kubernetes.io/projected/43ded71e-f4fa-4e66-a501-6f8792598ee2-kube-api-access-q69fx\") pod \"maas-api-revoke-sub-45de20f3-8ljw9\" (UID: \"43ded71e-f4fa-4e66-a501-6f8792598ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"
Oct 29 22:00:33.393192 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.393174    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45de20f3-8ljw9\" (UID: \"43ded71e-f4fa-4e66-a501-6f8792598ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"
Oct 29 22:00:33.399211 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.399188    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"default-dockercfg-8bczv\""
Oct 29 22:00:33.399317 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.399284    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"distinct-w0-e4e24e-kserve-self-signed-certs\""
Oct 29 22:00:33.431673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.427804    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"kube-root-ca.crt\""
Oct 29 22:00:33.431673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.427865    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"openshift-service-ca.crt\""
Oct 29 22:00:33.431673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.430793    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"]
Oct 29 22:00:33.496979 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.496940    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45de20f3-8ljw9\" (UID: \"43ded71e-f4fa-4e66-a501-6f8792598ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"
Oct 29 22:00:33.497162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.497012    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tmp-dir\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.497162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.497052    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-dshm\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.497162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.497131    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-model-cache\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.497318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.497294    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tls-certs\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.497458 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.497431    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbwn\" (UniqueName: \"kubernetes.io/projected/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-kube-api-access-4mbwn\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.497575 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.497492    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-home\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.497659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.497595    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45de20f3-8ljw9\" (UID: \"43ded71e-f4fa-4e66-a501-6f8792598ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"
Oct 29 22:00:33.497863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.497820    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q69fx\" (UniqueName: \"kubernetes.io/projected/43ded71e-f4fa-4e66-a501-6f8792598ee2-kube-api-access-q69fx\") pod \"maas-api-revoke-sub-45de20f3-8ljw9\" (UID: \"43ded71e-f4fa-4e66-a501-6f8792598ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"
Oct 29 22:00:33.498362 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.498340    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45de20f3-8ljw9\" (UID: \"43ded71e-f4fa-4e66-a501-6f8792598ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"
Oct 29 22:00:33.499493 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.499472    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45de20f3-8ljw9\" (UID: \"43ded71e-f4fa-4e66-a501-6f8792598ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"
Oct 29 22:00:33.524094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.524049    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q69fx\" (UniqueName: \"kubernetes.io/projected/43ded71e-f4fa-4e66-a501-6f8792598ee2-kube-api-access-q69fx\") pod \"maas-api-revoke-sub-45de20f3-8ljw9\" (UID: \"43ded71e-f4fa-4e66-a501-6f8792598ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"
Oct 29 22:00:33.558302 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.558265    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"
Oct 29 22:00:33.604654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.603747    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-home\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.604654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.603942    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tmp-dir\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.604654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.603980    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-dshm\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.604654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.604025    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-model-cache\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.604654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.604157    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tls-certs\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.604654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.604445    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbwn\" (UniqueName: \"kubernetes.io/projected/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-kube-api-access-4mbwn\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.633764 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.606670    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-home\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.633764 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.613368    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-dshm\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.633764 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.626131    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-model-cache\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.633764 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.626408    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tmp-dir\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.689650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.686530    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbwn\" (UniqueName: \"kubernetes.io/projected/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-kube-api-access-4mbwn\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.706186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.701377    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tls-certs\") pod \"distinct-w0-e4e24e-kserve-586d464fdd-bl4j7\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.706186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.705107    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:33.749651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.743383    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fcd3c41-6238-4a00-af83-becbfaef0e0b" path="/var/lib/kubelet/pods/0fcd3c41-6238-4a00-af83-becbfaef0e0b/volumes"
Oct 29 22:00:33.768344 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:33.767431    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f" path="/var/lib/kubelet/pods/e1f3f1b3-42aa-469c-a8e5-8360c5bcd18f/volumes"
Oct 29 22:00:34.241723 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:34.241694    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"]
Oct 29 22:00:34.252183 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:00:34.252150    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ded71e_f4fa_4e66_a501_6f8792598ee2.slice/crio-0f1ef5a0385ee69dd7e51ba674cf69d1cb0cedb17ccf9b850f9f293cda2a7113 WatchSource:0}: Error finding container 0f1ef5a0385ee69dd7e51ba674cf69d1cb0cedb17ccf9b850f9f293cda2a7113: Status 404 returned error can't find the container with id 0f1ef5a0385ee69dd7e51ba674cf69d1cb0cedb17ccf9b850f9f293cda2a7113
Oct 29 22:00:34.328677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:34.328649    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"
Oct 29 22:00:34.358065 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:00:34.358040    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf58ce73d_06fe_4f24_ada0_ba74bf58cb21.slice/crio-3bc1321e458c92e99294aec2180e36e35103466a7d6da693db89c605f0e81b0a WatchSource:0}: Error finding container 3bc1321e458c92e99294aec2180e36e35103466a7d6da693db89c605f0e81b0a: Status 404 returned error can't find the container with id 3bc1321e458c92e99294aec2180e36e35103466a7d6da693db89c605f0e81b0a
Oct 29 22:00:34.359267 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:34.359247    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"]
Oct 29 22:00:34.456216 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:34.456188    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f42909-c84a-49fa-97aa-d65fb90ff51b-kube-api-access-dw4cm\" (UniqueName: \"kubernetes.io/projected/02f42909-c84a-49fa-97aa-d65fb90ff51b-kube-api-access-dw4cm\") pod \"02f42909-c84a-49fa-97aa-d65fb90ff51b\" (UID: \"02f42909-c84a-49fa-97aa-d65fb90ff51b\") "
Oct 29 22:00:34.456360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:34.456341    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-openshift-ca\") pod \"02f42909-c84a-49fa-97aa-d65fb90ff51b\" (UID: \"02f42909-c84a-49fa-97aa-d65fb90ff51b\") "
Oct 29 22:00:34.456459 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:34.456442    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-xks-ca\") pod \"02f42909-c84a-49fa-97aa-d65fb90ff51b\" (UID: \"02f42909-c84a-49fa-97aa-d65fb90ff51b\") "
Oct 29 22:00:34.457546 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:34.457520    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-openshift-ca" pod "02f42909-c84a-49fa-97aa-d65fb90ff51b" (UID: "02f42909-c84a-49fa-97aa-d65fb90ff51b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:00:34.459392 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:34.459372    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f42909-c84a-49fa-97aa-d65fb90ff51b-kube-api-access-dw4cm" pod "02f42909-c84a-49fa-97aa-d65fb90ff51b" (UID: "02f42909-c84a-49fa-97aa-d65fb90ff51b"). InnerVolumeSpecName "kube-api-access-dw4cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:34.459467 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:34.459443    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-xks-ca" pod "02f42909-c84a-49fa-97aa-d65fb90ff51b" (UID: "02f42909-c84a-49fa-97aa-d65fb90ff51b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:00:34.561956 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:34.560814    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw4cm\" (UniqueName: \"kubernetes.io/projected/02f42909-c84a-49fa-97aa-d65fb90ff51b-kube-api-access-dw4cm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:34.561956 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:34.560851    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:34.561956 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:34.560865    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f42909-c84a-49fa-97aa-d65fb90ff51b-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:35.116118 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:35.115410    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9" event={"ID":"43ded71e-f4fa-4e66-a501-6f8792598ee2","Type":"ContainerStarted","Data":"0e609c0a42d4b8da1206ab479d4c042a2271477fb8e031085981a83fac102c03"}
Oct 29 22:00:35.116118 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:35.115489    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9" event={"ID":"43ded71e-f4fa-4e66-a501-6f8792598ee2","Type":"ContainerStarted","Data":"0f1ef5a0385ee69dd7e51ba674cf69d1cb0cedb17ccf9b850f9f293cda2a7113"}
Oct 29 22:00:35.117597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:35.117566    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d" event={"ID":"02f42909-c84a-49fa-97aa-d65fb90ff51b","Type":"ContainerDied","Data":"e8b53867914cf1a5cffb09d40c81c90d651af960e40b1e7e20b7399d5e8ec022"}
Oct 29 22:00:35.117597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:35.117581    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"
Oct 29 22:00:35.117806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:35.117612    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8b53867914cf1a5cffb09d40c81c90d651af960e40b1e7e20b7399d5e8ec022"
Oct 29 22:00:35.119713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:35.119686    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" event={"ID":"f58ce73d-06fe-4f24-ada0-ba74bf58cb21","Type":"ContainerStarted","Data":"bd1224e58549f8c494ec711a57d3adb091215cb2fca8e93c62e69ee1227bfe0b"}
Oct 29 22:00:35.119855 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:35.119723    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" event={"ID":"f58ce73d-06fe-4f24-ada0-ba74bf58cb21","Type":"ContainerStarted","Data":"3bc1321e458c92e99294aec2180e36e35103466a7d6da693db89c605f0e81b0a"}
Oct 29 22:00:35.159409 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:35.159366    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9" podStartSLOduration=2.159355154 podStartE2EDuration="2.159355154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:35.157010259 +0000 UTC m=+807.657052559" watchObservedRunningTime="2026-10-29 22:00:35.159355154 +0000 UTC m=+807.659397470"
Oct 29 22:00:35.201797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:35.201747    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podStartSLOduration=2.201733454 podStartE2EDuration="2.201733454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:35.199130444 +0000 UTC m=+807.699172744" watchObservedRunningTime="2026-10-29 22:00:35.201733454 +0000 UTC m=+807.701775754"
Oct 29 22:00:35.712080 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:35.711974    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:36.130231 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:36.130192    7082 generic.go:356] "Generic (PLEG): container finished" podID="43ded71e-f4fa-4e66-a501-6f8792598ee2" containerID="0e609c0a42d4b8da1206ab479d4c042a2271477fb8e031085981a83fac102c03" exitCode=0
Oct 29 22:00:36.130393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:36.130276    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9" event={"ID":"43ded71e-f4fa-4e66-a501-6f8792598ee2","Type":"ContainerDied","Data":"0e609c0a42d4b8da1206ab479d4c042a2271477fb8e031085981a83fac102c03"}
Oct 29 22:00:37.417557 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:37.417521    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"
Oct 29 22:00:37.537466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:37.537428    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-xks-ca\") pod \"43ded71e-f4fa-4e66-a501-6f8792598ee2\" (UID: \"43ded71e-f4fa-4e66-a501-6f8792598ee2\") "
Oct 29 22:00:37.537679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:37.537605    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ded71e-f4fa-4e66-a501-6f8792598ee2-kube-api-access-q69fx\" (UniqueName: \"kubernetes.io/projected/43ded71e-f4fa-4e66-a501-6f8792598ee2-kube-api-access-q69fx\") pod \"43ded71e-f4fa-4e66-a501-6f8792598ee2\" (UID: \"43ded71e-f4fa-4e66-a501-6f8792598ee2\") "
Oct 29 22:00:37.537883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:37.537863    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-openshift-ca\") pod \"43ded71e-f4fa-4e66-a501-6f8792598ee2\" (UID: \"43ded71e-f4fa-4e66-a501-6f8792598ee2\") "
Oct 29 22:00:37.538191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:37.538171    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-openshift-ca" pod "43ded71e-f4fa-4e66-a501-6f8792598ee2" (UID: "43ded71e-f4fa-4e66-a501-6f8792598ee2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:00:37.538547 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:37.538521    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:37.540278 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:37.540164    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ded71e-f4fa-4e66-a501-6f8792598ee2-kube-api-access-q69fx" pod "43ded71e-f4fa-4e66-a501-6f8792598ee2" (UID: "43ded71e-f4fa-4e66-a501-6f8792598ee2"). InnerVolumeSpecName "kube-api-access-q69fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:37.540278 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:37.540248    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-xks-ca" pod "43ded71e-f4fa-4e66-a501-6f8792598ee2" (UID: "43ded71e-f4fa-4e66-a501-6f8792598ee2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:00:37.639319 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:37.639290    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43ded71e-f4fa-4e66-a501-6f8792598ee2-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:37.639319 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:37.639315    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q69fx\" (UniqueName: \"kubernetes.io/projected/43ded71e-f4fa-4e66-a501-6f8792598ee2-kube-api-access-q69fx\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:38.149797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:38.149765    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"
Oct 29 22:00:38.149938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:38.149793    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9" event={"ID":"43ded71e-f4fa-4e66-a501-6f8792598ee2","Type":"ContainerDied","Data":"0f1ef5a0385ee69dd7e51ba674cf69d1cb0cedb17ccf9b850f9f293cda2a7113"}
Oct 29 22:00:38.149938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:38.149848    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f1ef5a0385ee69dd7e51ba674cf69d1cb0cedb17ccf9b850f9f293cda2a7113"
Oct 29 22:00:40.211173 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.211140    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"]
Oct 29 22:00:40.211930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.211911    7082 cpu_manager.go:414] "containerMap: removing container" podUID="02f42909-c84a-49fa-97aa-d65fb90ff51b" containerName="revoke-keys"
Oct 29 22:00:40.211930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.211930    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f42909-c84a-49fa-97aa-d65fb90ff51b" containerName="revoke-keys"
Oct 29 22:00:40.212077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.211974    7082 cpu_manager.go:414] "containerMap: removing container" podUID="43ded71e-f4fa-4e66-a501-6f8792598ee2" containerName="revoke-keys"
Oct 29 22:00:40.212077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.211979    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ded71e-f4fa-4e66-a501-6f8792598ee2" containerName="revoke-keys"
Oct 29 22:00:40.212180 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.212097    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ded71e-f4fa-4e66-a501-6f8792598ee2" containerName="revoke-keys"
Oct 29 22:00:40.212180 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.212110    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f42909-c84a-49fa-97aa-d65fb90ff51b" containerName="revoke-keys"
Oct 29 22:00:40.217124 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.217102    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.248354 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.248125    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"]
Oct 29 22:00:40.305393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.305356    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.305567 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.305551    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m6h2\" (UniqueName: \"kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-kube-api-access-5m6h2\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.305685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.305581    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.305786 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.305757    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.305786 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.305778    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.305907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.305880    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.305970 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.305921    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.306062 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.306045    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.306114 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.306086    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.407808 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.407777    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m6h2\" (UniqueName: \"kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-kube-api-access-5m6h2\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.407948 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.407817    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.407948 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.407918    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.407948 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.407939    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.408133 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.407974    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.408133 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.407995    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.408133 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.408093    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.408133 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.408128    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.408317 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.408183    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.408416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.408376    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.408867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.408674    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.408867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.408779    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.409199 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.408928    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.409199 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.409054    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.410530 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.410505    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.411055 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.411030    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.432549 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.432522    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.433078 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.433056    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m6h2\" (UniqueName: \"kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-kube-api-access-5m6h2\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.560668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.559974    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:40.943828 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.943803    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"]
Oct 29 22:00:40.944339 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:00:40.944316    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ff3c7f_c721_44e3_b83e_b214b93fc2b8.slice/crio-c0749db461789df6178c8dcecac6ed1b453d1e31c90c1c415bc3088b0ce07556 WatchSource:0}: Error finding container c0749db461789df6178c8dcecac6ed1b453d1e31c90c1c415bc3088b0ce07556: Status 404 returned error can't find the container with id c0749db461789df6178c8dcecac6ed1b453d1e31c90c1c415bc3088b0ce07556
Oct 29 22:00:40.946712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.946682    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:40.946770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.946744    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:40.946812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:40.946773    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:41.179101 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:41.179063    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" event={"ID":"85ff3c7f-c721-44e3-b83e-b214b93fc2b8","Type":"ContainerStarted","Data":"8d4f90c73ffefa4754a913a590dfe478c7d1d13715131a26c85853fca9f258d3"}
Oct 29 22:00:41.179246 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:41.179124    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" event={"ID":"85ff3c7f-c721-44e3-b83e-b214b93fc2b8","Type":"ContainerStarted","Data":"c0749db461789df6178c8dcecac6ed1b453d1e31c90c1c415bc3088b0ce07556"}
Oct 29 22:00:41.239873 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:41.239757    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" podStartSLOduration=1.239743079 podStartE2EDuration="1.239743079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:41.236784739 +0000 UTC m=+813.736827039" watchObservedRunningTime="2026-10-29 22:00:41.239743079 +0000 UTC m=+813.739785378"
Oct 29 22:00:41.562822 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:41.562723    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:42.562651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:42.562592    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 29 22:00:42.563162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:42.562679    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 29 22:00:43.560819 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:43.560785    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 29 22:00:43.561007 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:43.560845    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 29 22:00:43.715759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:43.715726    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:43.728000 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:43.727973    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:44.369183 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:44.369151    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:00:44.561568 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:44.561539    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 29 22:00:44.561737 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:44.561600    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 29 22:00:45.560890 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:45.560859    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 29 22:00:45.561287 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:45.560919    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 29 22:00:46.560751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:46.560709    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 29 22:00:46.560915 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:46.560783    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 29 22:00:47.560825 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:47.560779    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 29 22:00:47.561017 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:47.560854    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 29 22:00:48.565841 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:48.565805    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:48.566325 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:48.566301    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:48.567045 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:48.567023    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:00:50.579504 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.579472    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"]
Oct 29 22:00:50.584306 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.584284    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.636532 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.636483    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"]
Oct 29 22:00:50.744582 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.744546    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwxr4\" (UniqueName: \"kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-kube-api-access-bwxr4\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.744796 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.744705    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.744796 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.744741    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.744870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.744847    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.744918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.744906    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.745052 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.745027    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.745119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.745107    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.745151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.745135    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.745197 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.745187    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.846336 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.846238    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.846336 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.846311    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.846562 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.846348    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.846562 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.846403    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.846562 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.846503    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxr4\" (UniqueName: \"kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-kube-api-access-bwxr4\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.846766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.846596    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.846766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.846645    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.846766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.846729    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.846907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.846823    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.846907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.846816    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.846992 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.846965    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.847142 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.847119    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.847226 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.847208    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.847343 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.847319    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.848849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.848822    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.849366 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.849346    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.869951 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.869932    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.870092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.870076    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwxr4\" (UniqueName: \"kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-kube-api-access-bwxr4\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:50.941850 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:50.941817    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:51.351116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:51.351083    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"]
Oct 29 22:00:51.351472 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:00:51.351454    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1b8cff_4b0b_4b05_b2bc_2ea52741a114.slice/crio-b30b5eb737191e0cfa1c934e5887f2247fd40d55f733093675ec08fca9a5d7d5 WatchSource:0}: Error finding container b30b5eb737191e0cfa1c934e5887f2247fd40d55f733093675ec08fca9a5d7d5: Status 404 returned error can't find the container with id b30b5eb737191e0cfa1c934e5887f2247fd40d55f733093675ec08fca9a5d7d5
Oct 29 22:00:51.353901 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:51.353867    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:51.353974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:51.353947    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:51.354020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:51.353978    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:00:52.351399 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:52.351364    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" event={"ID":"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114","Type":"ContainerStarted","Data":"b7f92f156ecb8f1145be58a99a7e5443b9f50a5c0386018c29ce83fd21ccb0ad"}
Oct 29 22:00:52.351898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:52.351424    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" event={"ID":"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114","Type":"ContainerStarted","Data":"b30b5eb737191e0cfa1c934e5887f2247fd40d55f733093675ec08fca9a5d7d5"}
Oct 29 22:00:52.431234 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:52.431178    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" podStartSLOduration=2.431163722 podStartE2EDuration="2.431163722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:52.418102788 +0000 UTC m=+824.918145087" watchObservedRunningTime="2026-10-29 22:00:52.431163722 +0000 UTC m=+824.931206102"
Oct 29 22:00:52.946843 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:52.946810    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:52.952651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:52.949731    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:00:52.952651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:52.949793    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 29 22:00:53.942649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:53.942605    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:00:53.943203 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:53.942671    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 29 22:00:54.943129 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:54.943092    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:00:54.943530 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:54.943147    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 29 22:00:55.476139 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.476108    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"]
Oct 29 22:00:55.481685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.481666    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"
Oct 29 22:00:55.515653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.513940    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:00:55.515653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.513995    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:00:55.515653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.514034    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:00:55.515653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.514111    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:00:55.530211 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.529983    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"]
Oct 29 22:00:55.675672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.675595    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15541251-hh2s2\" (UID: \"163e7a73-075c-4531-9d07-21fd0de43a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"
Oct 29 22:00:55.675853 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.675717    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15541251-hh2s2\" (UID: \"163e7a73-075c-4531-9d07-21fd0de43a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"
Oct 29 22:00:55.675853 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.675751    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rcqg\" (UniqueName: \"kubernetes.io/projected/163e7a73-075c-4531-9d07-21fd0de43a97-kube-api-access-9rcqg\") pod \"maas-api-revoke-sub-15541251-hh2s2\" (UID: \"163e7a73-075c-4531-9d07-21fd0de43a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"
Oct 29 22:00:55.777082 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.777005    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15541251-hh2s2\" (UID: \"163e7a73-075c-4531-9d07-21fd0de43a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"
Oct 29 22:00:55.777282 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.777264    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15541251-hh2s2\" (UID: \"163e7a73-075c-4531-9d07-21fd0de43a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"
Oct 29 22:00:55.777368 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.777355    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rcqg\" (UniqueName: \"kubernetes.io/projected/163e7a73-075c-4531-9d07-21fd0de43a97-kube-api-access-9rcqg\") pod \"maas-api-revoke-sub-15541251-hh2s2\" (UID: \"163e7a73-075c-4531-9d07-21fd0de43a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"
Oct 29 22:00:55.777917 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.777893    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15541251-hh2s2\" (UID: \"163e7a73-075c-4531-9d07-21fd0de43a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"
Oct 29 22:00:55.779835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.779814    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15541251-hh2s2\" (UID: \"163e7a73-075c-4531-9d07-21fd0de43a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"
Oct 29 22:00:55.824589 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.824554    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rcqg\" (UniqueName: \"kubernetes.io/projected/163e7a73-075c-4531-9d07-21fd0de43a97-kube-api-access-9rcqg\") pod \"maas-api-revoke-sub-15541251-hh2s2\" (UID: \"163e7a73-075c-4531-9d07-21fd0de43a97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"
Oct 29 22:00:55.835360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.835332    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"
Oct 29 22:00:55.943863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.943373    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:00:55.943863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:55.943432    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 29 22:00:56.224575 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:56.224541    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"]
Oct 29 22:00:56.418881 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:56.418840    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2" event={"ID":"163e7a73-075c-4531-9d07-21fd0de43a97","Type":"ContainerStarted","Data":"4370edd813abbdcfd0d7de1c79fa2ced63f232a01ce801383b5ab94001a54bed"}
Oct 29 22:00:56.942976 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:56.942944    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:00:56.943149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:56.942994    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 29 22:00:57.029651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.029610    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"]
Oct 29 22:00:57.035403 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.035382    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"
Oct 29 22:00:57.069677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.069646    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"]
Oct 29 22:00:57.174893 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.174851    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aee2d99-55drw\" (UID: \"32abe6d7-0e48-4afd-9a10-be3881236679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"
Oct 29 22:00:57.175115 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.174997    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aee2d99-55drw\" (UID: \"32abe6d7-0e48-4afd-9a10-be3881236679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"
Oct 29 22:00:57.175269 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.175248    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmcg\" (UniqueName: \"kubernetes.io/projected/32abe6d7-0e48-4afd-9a10-be3881236679-kube-api-access-skmcg\") pod \"maas-api-revoke-sub-9aee2d99-55drw\" (UID: \"32abe6d7-0e48-4afd-9a10-be3881236679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"
Oct 29 22:00:57.277286 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.277194    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmcg\" (UniqueName: \"kubernetes.io/projected/32abe6d7-0e48-4afd-9a10-be3881236679-kube-api-access-skmcg\") pod \"maas-api-revoke-sub-9aee2d99-55drw\" (UID: \"32abe6d7-0e48-4afd-9a10-be3881236679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"
Oct 29 22:00:57.277286 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.277286    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aee2d99-55drw\" (UID: \"32abe6d7-0e48-4afd-9a10-be3881236679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"
Oct 29 22:00:57.277696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.277344    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aee2d99-55drw\" (UID: \"32abe6d7-0e48-4afd-9a10-be3881236679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"
Oct 29 22:00:57.277997 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.277971    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aee2d99-55drw\" (UID: \"32abe6d7-0e48-4afd-9a10-be3881236679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"
Oct 29 22:00:57.279845 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.279822    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aee2d99-55drw\" (UID: \"32abe6d7-0e48-4afd-9a10-be3881236679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"
Oct 29 22:00:57.315008 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.314745    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmcg\" (UniqueName: \"kubernetes.io/projected/32abe6d7-0e48-4afd-9a10-be3881236679-kube-api-access-skmcg\") pod \"maas-api-revoke-sub-9aee2d99-55drw\" (UID: \"32abe6d7-0e48-4afd-9a10-be3881236679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"
Oct 29 22:00:57.351223 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.351189    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"
Oct 29 22:00:57.449108 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.449074    7082 generic.go:356] "Generic (PLEG): container finished" podID="163e7a73-075c-4531-9d07-21fd0de43a97" containerID="27b48c33af7d9ddb4feb809c37af0ad272c12033d5b65c52268ab83158809847" exitCode=0
Oct 29 22:00:57.449260 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.449189    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2" event={"ID":"163e7a73-075c-4531-9d07-21fd0de43a97","Type":"ContainerDied","Data":"27b48c33af7d9ddb4feb809c37af0ad272c12033d5b65c52268ab83158809847"}
Oct 29 22:00:57.742050 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.742022    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"]
Oct 29 22:00:57.942805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.942773    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 29 22:00:57.943003 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:57.942825    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 29 22:00:58.502678 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:58.502545    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw" event={"ID":"32abe6d7-0e48-4afd-9a10-be3881236679","Type":"ContainerStarted","Data":"b9d349cdb8756357c9999c7bb611cd48516526d8e8ce76d08bfb2e9856a9930c"}
Oct 29 22:00:58.503673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:58.503608    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw" event={"ID":"32abe6d7-0e48-4afd-9a10-be3881236679","Type":"ContainerStarted","Data":"c5638e8134abb177baf49f8b712d239cfe6ec7468f3665c4a9fb20ab20825507"}
Oct 29 22:00:58.550917 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:00:58.550871    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:00:58.572359 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:58.571771    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw" podStartSLOduration=2.5717523289999997 podStartE2EDuration="2.571752329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:58.557316856 +0000 UTC m=+831.057359157" watchObservedRunningTime="2026-10-29 22:00:58.571752329 +0000 UTC m=+831.071794631"
Oct 29 22:00:58.710906 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:58.710875    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"
Oct 29 22:00:58.751944 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:58.751906    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163e7a73-075c-4531-9d07-21fd0de43a97-kube-api-access-9rcqg\" (UniqueName: \"kubernetes.io/projected/163e7a73-075c-4531-9d07-21fd0de43a97-kube-api-access-9rcqg\") pod \"163e7a73-075c-4531-9d07-21fd0de43a97\" (UID: \"163e7a73-075c-4531-9d07-21fd0de43a97\") "
Oct 29 22:00:58.752157 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:58.752004    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-openshift-ca\") pod \"163e7a73-075c-4531-9d07-21fd0de43a97\" (UID: \"163e7a73-075c-4531-9d07-21fd0de43a97\") "
Oct 29 22:00:58.752157 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:58.752032    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-xks-ca\") pod \"163e7a73-075c-4531-9d07-21fd0de43a97\" (UID: \"163e7a73-075c-4531-9d07-21fd0de43a97\") "
Oct 29 22:00:58.752935 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:58.752730    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-openshift-ca" pod "163e7a73-075c-4531-9d07-21fd0de43a97" (UID: "163e7a73-075c-4531-9d07-21fd0de43a97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:00:58.753292 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:58.753087    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:58.755054 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:58.755013    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-xks-ca" pod "163e7a73-075c-4531-9d07-21fd0de43a97" (UID: "163e7a73-075c-4531-9d07-21fd0de43a97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:00:58.755679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:58.755647    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163e7a73-075c-4531-9d07-21fd0de43a97-kube-api-access-9rcqg" pod "163e7a73-075c-4531-9d07-21fd0de43a97" (UID: "163e7a73-075c-4531-9d07-21fd0de43a97"). InnerVolumeSpecName "kube-api-access-9rcqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:58.854498 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:58.854449    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rcqg\" (UniqueName: \"kubernetes.io/projected/163e7a73-075c-4531-9d07-21fd0de43a97-kube-api-access-9rcqg\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:58.854498 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:58.854484    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163e7a73-075c-4531-9d07-21fd0de43a97-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:00:59.108183 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:59.108149    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:59.110527 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:59.109768    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:59.111150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:59.110912    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:00:59.511885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:59.511811    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"
Oct 29 22:00:59.511885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:59.511848    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2" event={"ID":"163e7a73-075c-4531-9d07-21fd0de43a97","Type":"ContainerDied","Data":"4370edd813abbdcfd0d7de1c79fa2ced63f232a01ce801383b5ab94001a54bed"}
Oct 29 22:00:59.512385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:59.511901    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4370edd813abbdcfd0d7de1c79fa2ced63f232a01ce801383b5ab94001a54bed"
Oct 29 22:00:59.513700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:59.513673    7082 generic.go:356] "Generic (PLEG): container finished" podID="32abe6d7-0e48-4afd-9a10-be3881236679" containerID="b9d349cdb8756357c9999c7bb611cd48516526d8e8ce76d08bfb2e9856a9930c" exitCode=0
Oct 29 22:00:59.513859 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:00:59.513743    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw" event={"ID":"32abe6d7-0e48-4afd-9a10-be3881236679","Type":"ContainerDied","Data":"b9d349cdb8756357c9999c7bb611cd48516526d8e8ce76d08bfb2e9856a9930c"}
Oct 29 22:01:00.704746 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:00.704712    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"
Oct 29 22:01:00.788704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:00.788561    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-openshift-ca\") pod \"32abe6d7-0e48-4afd-9a10-be3881236679\" (UID: \"32abe6d7-0e48-4afd-9a10-be3881236679\") "
Oct 29 22:01:00.788704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:00.788689    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-xks-ca\") pod \"32abe6d7-0e48-4afd-9a10-be3881236679\" (UID: \"32abe6d7-0e48-4afd-9a10-be3881236679\") "
Oct 29 22:01:00.788938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:00.788882    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32abe6d7-0e48-4afd-9a10-be3881236679-kube-api-access-skmcg\" (UniqueName: \"kubernetes.io/projected/32abe6d7-0e48-4afd-9a10-be3881236679-kube-api-access-skmcg\") pod \"32abe6d7-0e48-4afd-9a10-be3881236679\" (UID: \"32abe6d7-0e48-4afd-9a10-be3881236679\") "
Oct 29 22:01:00.789005 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:00.788983    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-openshift-ca" pod "32abe6d7-0e48-4afd-9a10-be3881236679" (UID: "32abe6d7-0e48-4afd-9a10-be3881236679"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:01:00.789920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:00.789892    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:00.791503 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:00.791469    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-xks-ca" pod "32abe6d7-0e48-4afd-9a10-be3881236679" (UID: "32abe6d7-0e48-4afd-9a10-be3881236679"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:01:00.792122 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:00.792083    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32abe6d7-0e48-4afd-9a10-be3881236679-kube-api-access-skmcg" pod "32abe6d7-0e48-4afd-9a10-be3881236679" (UID: "32abe6d7-0e48-4afd-9a10-be3881236679"). InnerVolumeSpecName "kube-api-access-skmcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:01:00.891664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:00.891609    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32abe6d7-0e48-4afd-9a10-be3881236679-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:00.891664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:00.891662    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skmcg\" (UniqueName: \"kubernetes.io/projected/32abe6d7-0e48-4afd-9a10-be3881236679-kube-api-access-skmcg\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:01.533067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:01.533033    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"
Oct 29 22:01:01.533067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:01.533054    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw" event={"ID":"32abe6d7-0e48-4afd-9a10-be3881236679","Type":"ContainerDied","Data":"c5638e8134abb177baf49f8b712d239cfe6ec7468f3665c4a9fb20ab20825507"}
Oct 29 22:01:01.533293 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:01.533117    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5638e8134abb177baf49f8b712d239cfe6ec7468f3665c4a9fb20ab20825507"
Oct 29 22:01:08.151983 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.151949    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"]
Oct 29 22:01:08.152706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.152691    7082 cpu_manager.go:414] "containerMap: removing container" podUID="163e7a73-075c-4531-9d07-21fd0de43a97" containerName="revoke-keys"
Oct 29 22:01:08.152752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.152707    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163e7a73-075c-4531-9d07-21fd0de43a97" containerName="revoke-keys"
Oct 29 22:01:08.152752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.152723    7082 cpu_manager.go:414] "containerMap: removing container" podUID="32abe6d7-0e48-4afd-9a10-be3881236679" containerName="revoke-keys"
Oct 29 22:01:08.152752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.152728    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32abe6d7-0e48-4afd-9a10-be3881236679" containerName="revoke-keys"
Oct 29 22:01:08.152851 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.152825    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="163e7a73-075c-4531-9d07-21fd0de43a97" containerName="revoke-keys"
Oct 29 22:01:08.152851 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.152836    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="32abe6d7-0e48-4afd-9a10-be3881236679" containerName="revoke-keys"
Oct 29 22:01:08.157942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.157923    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"
Oct 29 22:01:08.175958 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.175931    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"]
Oct 29 22:01:08.182252 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.182219    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:01:08.182496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.182291    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:01:08.182496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.182379    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:01:08.184431 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.183205    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:01:08.265712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.265685    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr28s\" (UniqueName: \"kubernetes.io/projected/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-kube-api-access-qr28s\") pod \"maas-api-revoke-sub-81a077d7-vbrt9\" (UID: \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"
Oct 29 22:01:08.265913 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.265900    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81a077d7-vbrt9\" (UID: \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"
Oct 29 22:01:08.265968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.265944    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81a077d7-vbrt9\" (UID: \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"
Oct 29 22:01:08.367426 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.367385    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr28s\" (UniqueName: \"kubernetes.io/projected/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-kube-api-access-qr28s\") pod \"maas-api-revoke-sub-81a077d7-vbrt9\" (UID: \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"
Oct 29 22:01:08.367603 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.367564    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81a077d7-vbrt9\" (UID: \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"
Oct 29 22:01:08.367666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.367600    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81a077d7-vbrt9\" (UID: \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"
Oct 29 22:01:08.368397 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.368372    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81a077d7-vbrt9\" (UID: \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"
Oct 29 22:01:08.370209 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.370187    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81a077d7-vbrt9\" (UID: \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"
Oct 29 22:01:08.391909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.391886    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr28s\" (UniqueName: \"kubernetes.io/projected/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-kube-api-access-qr28s\") pod \"maas-api-revoke-sub-81a077d7-vbrt9\" (UID: \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"
Oct 29 22:01:08.474516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.474439    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"
Oct 29 22:01:08.789521 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:01:08.789488    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d7106f_f979_43d1_b91e_4ed51f5d2bf8.slice/crio-b14c2a6d98878a40d12feefc7f48d1d81db3fe53c185f3db4e2d733e158d81ec WatchSource:0}: Error finding container b14c2a6d98878a40d12feefc7f48d1d81db3fe53c185f3db4e2d733e158d81ec: Status 404 returned error can't find the container with id b14c2a6d98878a40d12feefc7f48d1d81db3fe53c185f3db4e2d733e158d81ec
Oct 29 22:01:08.791548 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:08.791372    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"]
Oct 29 22:01:09.650863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:09.650823    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9" event={"ID":"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8","Type":"ContainerStarted","Data":"70149604df4e993c107e5d932e728d3429765951206a62b67b09e3b98943f076"}
Oct 29 22:01:09.651303 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:09.650883    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9" event={"ID":"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8","Type":"ContainerStarted","Data":"b14c2a6d98878a40d12feefc7f48d1d81db3fe53c185f3db4e2d733e158d81ec"}
Oct 29 22:01:09.694259 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:09.694195    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9" podStartSLOduration=1.694178985 podStartE2EDuration="1.694178985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:09.693411637 +0000 UTC m=+842.193453949" watchObservedRunningTime="2026-10-29 22:01:09.694178985 +0000 UTC m=+842.194221283"
Oct 29 22:01:10.707647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:10.705557    7082 generic.go:356] "Generic (PLEG): container finished" podID="c6d7106f-f979-43d1-b91e-4ed51f5d2bf8" containerID="70149604df4e993c107e5d932e728d3429765951206a62b67b09e3b98943f076" exitCode=0
Oct 29 22:01:10.707647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:10.705815    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9" event={"ID":"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8","Type":"ContainerDied","Data":"70149604df4e993c107e5d932e728d3429765951206a62b67b09e3b98943f076"}
Oct 29 22:01:11.880710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:11.880683    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"
Oct 29 22:01:11.974836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:11.974744    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-kube-api-access-qr28s\" (UniqueName: \"kubernetes.io/projected/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-kube-api-access-qr28s\") pod \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\" (UID: \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\") "
Oct 29 22:01:11.975004 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:11.974877    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-xks-ca\") pod \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\" (UID: \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\") "
Oct 29 22:01:11.975004 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:11.974922    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-openshift-ca\") pod \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\" (UID: \"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8\") "
Oct 29 22:01:11.975711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:11.975620    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-openshift-ca" pod "c6d7106f-f979-43d1-b91e-4ed51f5d2bf8" (UID: "c6d7106f-f979-43d1-b91e-4ed51f5d2bf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:01:11.976054 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:11.976030    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:11.977356 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:11.977332    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-xks-ca" pod "c6d7106f-f979-43d1-b91e-4ed51f5d2bf8" (UID: "c6d7106f-f979-43d1-b91e-4ed51f5d2bf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:01:11.977577 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:11.977554    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-kube-api-access-qr28s" pod "c6d7106f-f979-43d1-b91e-4ed51f5d2bf8" (UID: "c6d7106f-f979-43d1-b91e-4ed51f5d2bf8"). InnerVolumeSpecName "kube-api-access-qr28s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:01:12.069053 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.069020    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"]
Oct 29 22:01:12.069773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.069758    7082 cpu_manager.go:414] "containerMap: removing container" podUID="c6d7106f-f979-43d1-b91e-4ed51f5d2bf8" containerName="revoke-keys"
Oct 29 22:01:12.069845 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.069774    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d7106f-f979-43d1-b91e-4ed51f5d2bf8" containerName="revoke-keys"
Oct 29 22:01:12.069932 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.069923    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d7106f-f979-43d1-b91e-4ed51f5d2bf8" containerName="revoke-keys"
Oct 29 22:01:12.074786 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.074752    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"
Oct 29 22:01:12.077711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.077684    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr28s\" (UniqueName: \"kubernetes.io/projected/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-kube-api-access-qr28s\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:12.077841 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.077714    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:12.102772 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.100085    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"]
Oct 29 22:01:12.179375 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.179338    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mljjn\" (UniqueName: \"kubernetes.io/projected/f3e98faa-9f44-41f2-af30-47a0260f1ab1-kube-api-access-mljjn\") pod \"maas-api-revoke-sub-7cffbe85-hx5zr\" (UID: \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"
Oct 29 22:01:12.179562 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.179543    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cffbe85-hx5zr\" (UID: \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"
Oct 29 22:01:12.179613 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.179602    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cffbe85-hx5zr\" (UID: \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"
Oct 29 22:01:12.281276 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.281184    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cffbe85-hx5zr\" (UID: \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"
Oct 29 22:01:12.281276 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.281268    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cffbe85-hx5zr\" (UID: \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"
Oct 29 22:01:12.281544 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.281529    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mljjn\" (UniqueName: \"kubernetes.io/projected/f3e98faa-9f44-41f2-af30-47a0260f1ab1-kube-api-access-mljjn\") pod \"maas-api-revoke-sub-7cffbe85-hx5zr\" (UID: \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"
Oct 29 22:01:12.282003 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.281971    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cffbe85-hx5zr\" (UID: \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"
Oct 29 22:01:12.283764 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.283747    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cffbe85-hx5zr\" (UID: \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"
Oct 29 22:01:12.306670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.306650    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mljjn\" (UniqueName: \"kubernetes.io/projected/f3e98faa-9f44-41f2-af30-47a0260f1ab1-kube-api-access-mljjn\") pod \"maas-api-revoke-sub-7cffbe85-hx5zr\" (UID: \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"
Oct 29 22:01:12.394580 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.394546    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"
Oct 29 22:01:12.723991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.723937    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"
Oct 29 22:01:12.723991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.723977    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9" event={"ID":"c6d7106f-f979-43d1-b91e-4ed51f5d2bf8","Type":"ContainerDied","Data":"b14c2a6d98878a40d12feefc7f48d1d81db3fe53c185f3db4e2d733e158d81ec"}
Oct 29 22:01:12.724321 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.724039    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b14c2a6d98878a40d12feefc7f48d1d81db3fe53c185f3db4e2d733e158d81ec"
Oct 29 22:01:12.750902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:12.750876    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"]
Oct 29 22:01:12.751322 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:01:12.751298    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e98faa_9f44_41f2_af30_47a0260f1ab1.slice/crio-0c87f0dcfd73e224a78cb2c74bd9f086d5c2440ebd0f257611b4741349bbeef8 WatchSource:0}: Error finding container 0c87f0dcfd73e224a78cb2c74bd9f086d5c2440ebd0f257611b4741349bbeef8: Status 404 returned error can't find the container with id 0c87f0dcfd73e224a78cb2c74bd9f086d5c2440ebd0f257611b4741349bbeef8
Oct 29 22:01:13.734841 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:13.734803    7082 generic.go:356] "Generic (PLEG): container finished" podID="f3e98faa-9f44-41f2-af30-47a0260f1ab1" containerID="b4e67939aad355f6758ce00510b08f1f9267adfae65ce7bdb7d6f59abf660b11" exitCode=0
Oct 29 22:01:13.735256 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:13.734842    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr" event={"ID":"f3e98faa-9f44-41f2-af30-47a0260f1ab1","Type":"ContainerDied","Data":"b4e67939aad355f6758ce00510b08f1f9267adfae65ce7bdb7d6f59abf660b11"}
Oct 29 22:01:13.735256 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:13.734891    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr" event={"ID":"f3e98faa-9f44-41f2-af30-47a0260f1ab1","Type":"ContainerStarted","Data":"0c87f0dcfd73e224a78cb2c74bd9f086d5c2440ebd0f257611b4741349bbeef8"}
Oct 29 22:01:14.928020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:14.927996    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"
Oct 29 22:01:15.054701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:15.054663    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e98faa-9f44-41f2-af30-47a0260f1ab1-kube-api-access-mljjn\" (UniqueName: \"kubernetes.io/projected/f3e98faa-9f44-41f2-af30-47a0260f1ab1-kube-api-access-mljjn\") pod \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\" (UID: \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\") "
Oct 29 22:01:15.054913 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:15.054709    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-xks-ca\") pod \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\" (UID: \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\") "
Oct 29 22:01:15.054913 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:15.054768    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-openshift-ca\") pod \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\" (UID: \"f3e98faa-9f44-41f2-af30-47a0260f1ab1\") "
Oct 29 22:01:15.055238 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:15.055206    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-openshift-ca" pod "f3e98faa-9f44-41f2-af30-47a0260f1ab1" (UID: "f3e98faa-9f44-41f2-af30-47a0260f1ab1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:01:15.055573 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:15.055553    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:15.057088 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:15.057063    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-xks-ca" pod "f3e98faa-9f44-41f2-af30-47a0260f1ab1" (UID: "f3e98faa-9f44-41f2-af30-47a0260f1ab1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:01:15.057404 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:15.057378    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e98faa-9f44-41f2-af30-47a0260f1ab1-kube-api-access-mljjn" pod "f3e98faa-9f44-41f2-af30-47a0260f1ab1" (UID: "f3e98faa-9f44-41f2-af30-47a0260f1ab1"). InnerVolumeSpecName "kube-api-access-mljjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:01:15.156768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:15.156735    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mljjn\" (UniqueName: \"kubernetes.io/projected/f3e98faa-9f44-41f2-af30-47a0260f1ab1-kube-api-access-mljjn\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:15.156768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:15.156764    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e98faa-9f44-41f2-af30-47a0260f1ab1-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:15.753075 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:15.753042    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"
Oct 29 22:01:15.753236 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:15.753072    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr" event={"ID":"f3e98faa-9f44-41f2-af30-47a0260f1ab1","Type":"ContainerDied","Data":"0c87f0dcfd73e224a78cb2c74bd9f086d5c2440ebd0f257611b4741349bbeef8"}
Oct 29 22:01:15.753236 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:15.753117    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c87f0dcfd73e224a78cb2c74bd9f086d5c2440ebd0f257611b4741349bbeef8"
Oct 29 22:01:21.846570 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:21.846533    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:01:26.990585 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:26.989160    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:27.260044 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:27.259914    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:27.392654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:27.376743    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:27.392654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:27.379849    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:01:27.480797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:27.480734    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:27.868651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:27.854789    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:27.999789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:27.999733    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.002085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:28.002054    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:01:28.049910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:28.049844    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.364711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:28.313741    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.364711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:28.314513    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:01:28.392680 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:28.379855    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.392680 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:28.380163    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.409658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:28.401401    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:01:28.523702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:28.413713    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:01:28.523702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:28.414159    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:01:28.523702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:28.481541    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.523702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:28.482109    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:01:29.072655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:29.037750    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.146654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:29.084495    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:01:29.156588 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:29.153917    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:01:29.314611 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:29.314569    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.393707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:29.382258    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.418333 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:29.418300    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:01:29.419833 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:29.391871    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.421039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:29.420995    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.427787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:29.427750    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.487983 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:29.487921    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.841309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:29.815366    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.063911 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:30.062740    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:01:30.153884 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:30.153763    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:01:30.344549 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:30.337294    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:01:30.427715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:30.414998    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.427715 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:01:30.415833    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:01:30.427715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:30.423752    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:01:30.427715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:30.424298    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.523653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:30.523490    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.113651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:31.081277    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.363503 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:31.318783    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.437109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:31.437003    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.500683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:31.500560    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.501665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:31.367754    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:01:31.510715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:31.418033    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.678081 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:31.603668    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.178743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:32.178485    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Oct 29 22:01:32.348676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:32.346852    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.424656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:32.424087    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:01:32.462052 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:32.442915    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.552686 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:32.526712    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:01:32.552686 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:32.550014    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:01:32.755884 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:32.755767    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.786758 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:32.786688    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.348553 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:33.348511    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.428545 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:33.428497    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.444797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:33.444746    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.535780 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:33.535727    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.561000 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:33.560952    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:01:33.582874 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:33.582827    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.791715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:33.784840    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:01:33.801655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:33.793222    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.735914 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.718954    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:34.735914 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.719026    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.735914 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.733284    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.735914 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.733742    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.780949 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.772853    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:01:34.780949 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.774573    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.802660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.797878    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:34.802660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.797954    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.818713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.815935    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:01:34.846705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.845878    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:34.846705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.845967    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.860186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.860144    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:34.860503 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.860459    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.866655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.860704    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:01:34.873658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.867261    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:01:34.935654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.875061    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": context deadline exceeded"
Oct 29 22:01:34.979654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.914844    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:34.979654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.962845    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.979654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.936186    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:34.979654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.962931    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.985982 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.985881    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:34.986258 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:34.986223    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.097819 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:35.077197    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.790790 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:35.789131    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.817366 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:35.817314    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:01:35.819087 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:35.819027    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:01:35.819452 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:35.819402    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.825970 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:35.825929    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.914684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:35.911223    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.970797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:35.946352    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:01:36.810694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:36.809802    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:36.848171 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:36.847269    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:36.886449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:36.856581    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:36.886449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:36.856916    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:01:36.886449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:36.858536    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:36.911594 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:36.911547    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:37.052789 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:01:37.045230    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.402s"
Oct 29 22:01:37.068971 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:37.068866    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:37.925329 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:37.872275    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:37.948887 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:37.948818    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:01:37.950787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:37.950754    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:37.951059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:37.951011    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:37.992791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:37.973998    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:01:38.014617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:38.014571    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:38.014935 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:38.014889    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" podUID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:38.015090 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:37.974125    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:01:38.016463 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:37.974089    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded"
Oct 29 22:01:38.016664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:37.974147    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:01:38.093888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:37.974067    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:01:38.100826 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:38.100778    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:38.878690 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:38.875718    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:38.961664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:38.961596    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:39.002693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:38.996718    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:01:39.018879 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:39.018826    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:39.064654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:39.053918    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:39.097716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:39.094611    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:39.150706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:39.124807    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:39.424917 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:39.424777    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:39.553691 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:39.506247    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:01:39.928977 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:39.928947    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:01:39.963693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:39.963612    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:01:39.975248 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:39.973751    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:40.002140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:40.002107    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:01:40.020660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:40.019190    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:01:40.037203 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:40.036730    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:40.062012 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:40.061969    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:01:40.067080 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:40.067044    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:01:40.159095 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:40.159040    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:01:43.634938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.634908    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"]
Oct 29 22:01:43.635667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.635638    7082 cpu_manager.go:414] "containerMap: removing container" podUID="f3e98faa-9f44-41f2-af30-47a0260f1ab1" containerName="revoke-keys"
Oct 29 22:01:43.635667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.635654    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e98faa-9f44-41f2-af30-47a0260f1ab1" containerName="revoke-keys"
Oct 29 22:01:43.635819 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.635809    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e98faa-9f44-41f2-af30-47a0260f1ab1" containerName="revoke-keys"
Oct 29 22:01:43.639847 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.639828    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"
Oct 29 22:01:43.663675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.653836    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"]
Oct 29 22:01:43.663675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.656363    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:01:43.663675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.658674    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:01:43.663675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.658745    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:01:43.668212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.668188    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:01:43.818571 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.818538    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d4314b-h5j2f\" (UID: \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"
Oct 29 22:01:43.818789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.818612    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpdgl\" (UniqueName: \"kubernetes.io/projected/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-kube-api-access-bpdgl\") pod \"maas-api-revoke-sub-24d4314b-h5j2f\" (UID: \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"
Oct 29 22:01:43.818849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.818789    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d4314b-h5j2f\" (UID: \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"
Oct 29 22:01:43.920949 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.920858    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d4314b-h5j2f\" (UID: \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"
Oct 29 22:01:43.921115 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.920957    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpdgl\" (UniqueName: \"kubernetes.io/projected/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-kube-api-access-bpdgl\") pod \"maas-api-revoke-sub-24d4314b-h5j2f\" (UID: \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"
Oct 29 22:01:43.921115 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.921015    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d4314b-h5j2f\" (UID: \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"
Oct 29 22:01:43.922104 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.922041    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d4314b-h5j2f\" (UID: \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"
Oct 29 22:01:43.934418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.934302    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d4314b-h5j2f\" (UID: \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"
Oct 29 22:01:43.945035 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.945007    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpdgl\" (UniqueName: \"kubernetes.io/projected/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-kube-api-access-bpdgl\") pod \"maas-api-revoke-sub-24d4314b-h5j2f\" (UID: \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"
Oct 29 22:01:43.960259 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:43.960233    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"
Oct 29 22:01:44.226161 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:44.226072    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"]
Oct 29 22:01:44.226333 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:44.226279    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j" podUID="1644588d-6cb4-4f39-ab4a-04e939a89fe6" containerName="istio-proxy" containerID="cri-o://ba50b1f9d11ae1f1988c187c47c1dbb3e6fc705c06d2009883df074a4c324d48" gracePeriod=30
Oct 29 22:01:44.290839 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:44.290806    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"]
Oct 29 22:01:44.738874 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:44.738833    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f" event={"ID":"eb3e1fca-d76f-4fba-9a59-2cb8600aefda","Type":"ContainerStarted","Data":"8e3b4ead1cb25ff086abe1b94e81d1bed1301c681d943b6a18f637d6911157a3"}
Oct 29 22:01:44.739318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:44.738900    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f" event={"ID":"eb3e1fca-d76f-4fba-9a59-2cb8600aefda","Type":"ContainerStarted","Data":"5776d7c5650b0e301c57cafc0f5d0d5f745b4a314b059c2b7f71400cc5abfd8f"}
Oct 29 22:01:44.784803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:44.784702    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f" podStartSLOduration=1.784688118 podStartE2EDuration="1.784688118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:44.7800347 +0000 UTC m=+877.280077004" watchObservedRunningTime="2026-10-29 22:01:44.784688118 +0000 UTC m=+877.284730432"
Oct 29 22:01:45.752119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:45.752000    7082 generic.go:356] "Generic (PLEG): container finished" podID="eb3e1fca-d76f-4fba-9a59-2cb8600aefda" containerID="8e3b4ead1cb25ff086abe1b94e81d1bed1301c681d943b6a18f637d6911157a3" exitCode=0
Oct 29 22:01:45.752119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:45.752074    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f" event={"ID":"eb3e1fca-d76f-4fba-9a59-2cb8600aefda","Type":"ContainerDied","Data":"8e3b4ead1cb25ff086abe1b94e81d1bed1301c681d943b6a18f637d6911157a3"}
Oct 29 22:01:46.944559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:46.944529    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"
Oct 29 22:01:47.051369 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:47.051323    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-openshift-ca\") pod \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\" (UID: \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\") "
Oct 29 22:01:47.051798 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:47.051779    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-kube-api-access-bpdgl\" (UniqueName: \"kubernetes.io/projected/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-kube-api-access-bpdgl\") pod \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\" (UID: \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\") "
Oct 29 22:01:47.051985 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:47.051793    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-openshift-ca" pod "eb3e1fca-d76f-4fba-9a59-2cb8600aefda" (UID: "eb3e1fca-d76f-4fba-9a59-2cb8600aefda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:01:47.052385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:47.052365    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-xks-ca\") pod \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\" (UID: \"eb3e1fca-d76f-4fba-9a59-2cb8600aefda\") "
Oct 29 22:01:47.054388 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:47.054364    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:47.057758 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:47.057120    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-kube-api-access-bpdgl" pod "eb3e1fca-d76f-4fba-9a59-2cb8600aefda" (UID: "eb3e1fca-d76f-4fba-9a59-2cb8600aefda"). InnerVolumeSpecName "kube-api-access-bpdgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:01:47.058133 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:47.058110    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-xks-ca" pod "eb3e1fca-d76f-4fba-9a59-2cb8600aefda" (UID: "eb3e1fca-d76f-4fba-9a59-2cb8600aefda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:01:47.156810 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:47.156773    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:47.156810 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:47.156811    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpdgl\" (UniqueName: \"kubernetes.io/projected/eb3e1fca-d76f-4fba-9a59-2cb8600aefda-kube-api-access-bpdgl\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:47.853018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:47.852930    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f" event={"ID":"eb3e1fca-d76f-4fba-9a59-2cb8600aefda","Type":"ContainerDied","Data":"5776d7c5650b0e301c57cafc0f5d0d5f745b4a314b059c2b7f71400cc5abfd8f"}
Oct 29 22:01:47.853424 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:47.853399    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5776d7c5650b0e301c57cafc0f5d0d5f745b4a314b059c2b7f71400cc5abfd8f"
Oct 29 22:01:47.853700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:47.853685    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"
Oct 29 22:01:49.610917 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.610891    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 22:01:49.719658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.719609    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istiod-ca-cert\") pod \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") "
Oct 29 22:01:49.719849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.719721    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-credential-socket\") pod \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") "
Oct 29 22:01:49.719929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.719856    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-envoy\") pod \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") "
Oct 29 22:01:49.719929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.719909    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-podinfo\") pod \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") "
Oct 29 22:01:49.720037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.719949    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-token\" (UniqueName: \"kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-token\") pod \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") "
Oct 29 22:01:49.720037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.720003    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-certs\") pod \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") "
Oct 29 22:01:49.720149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.720047    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-credential-socket" pod "1644588d-6cb4-4f39-ab4a-04e939a89fe6" (UID: "1644588d-6cb4-4f39-ab4a-04e939a89fe6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:49.720149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.720091    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istiod-ca-cert" pod "1644588d-6cb4-4f39-ab4a-04e939a89fe6" (UID: "1644588d-6cb4-4f39-ab4a-04e939a89fe6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:01:49.720149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.720136    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-data\") pod \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") "
Oct 29 22:01:49.720398 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.720234    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-certs" pod "1644588d-6cb4-4f39-ab4a-04e939a89fe6" (UID: "1644588d-6cb4-4f39-ab4a-04e939a89fe6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:49.720398 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.720253    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-kube-api-access-7qcjt\" (UniqueName: \"kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-kube-api-access-7qcjt\") pod \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") "
Oct 29 22:01:49.720398 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.720370    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-socket\") pod \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\" (UID: \"1644588d-6cb4-4f39-ab4a-04e939a89fe6\") "
Oct 29 22:01:49.720558 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.720410    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-data" pod "1644588d-6cb4-4f39-ab4a-04e939a89fe6" (UID: "1644588d-6cb4-4f39-ab4a-04e939a89fe6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:49.720759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.720737    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-socket" pod "1644588d-6cb4-4f39-ab4a-04e939a89fe6" (UID: "1644588d-6cb4-4f39-ab4a-04e939a89fe6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:49.721712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.721389    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:49.721712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.721415    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:49.721712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.721428    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:49.721712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.721442    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:49.721712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.721455    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:49.723086 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.723065    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-token" pod "1644588d-6cb4-4f39-ab4a-04e939a89fe6" (UID: "1644588d-6cb4-4f39-ab4a-04e939a89fe6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:01:49.723294 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.723122    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-envoy" pod "1644588d-6cb4-4f39-ab4a-04e939a89fe6" (UID: "1644588d-6cb4-4f39-ab4a-04e939a89fe6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:49.724168 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.724138    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-kube-api-access-7qcjt" pod "1644588d-6cb4-4f39-ab4a-04e939a89fe6" (UID: "1644588d-6cb4-4f39-ab4a-04e939a89fe6"). InnerVolumeSpecName "kube-api-access-7qcjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:01:49.724534 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.724514    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-podinfo" pod "1644588d-6cb4-4f39-ab4a-04e939a89fe6" (UID: "1644588d-6cb4-4f39-ab4a-04e939a89fe6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:01:49.822916 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.822884    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:49.822916 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.822915    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:49.822916 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.822925    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:49.823138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.822933    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qcjt\" (UniqueName: \"kubernetes.io/projected/1644588d-6cb4-4f39-ab4a-04e939a89fe6-kube-api-access-7qcjt\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:01:49.869761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.869678    7082 generic.go:356] "Generic (PLEG): container finished" podID="1644588d-6cb4-4f39-ab4a-04e939a89fe6" containerID="ba50b1f9d11ae1f1988c187c47c1dbb3e6fc705c06d2009883df074a4c324d48" exitCode=0
Oct 29 22:01:49.869905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.869763    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"
Oct 29 22:01:49.869905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.869802    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j" event={"ID":"1644588d-6cb4-4f39-ab4a-04e939a89fe6","Type":"ContainerDied","Data":"ba50b1f9d11ae1f1988c187c47c1dbb3e6fc705c06d2009883df074a4c324d48"}
Oct 29 22:01:49.869905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.869865    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j" event={"ID":"1644588d-6cb4-4f39-ab4a-04e939a89fe6","Type":"ContainerDied","Data":"41617c4f446b5bbc229946f176fb028b02e69e0d2682ff124bafd9fc2a96ee45"}
Oct 29 22:01:49.869905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.869888    7082 scope.go:122] "RemoveContainer" containerID="ba50b1f9d11ae1f1988c187c47c1dbb3e6fc705c06d2009883df074a4c324d48"
Oct 29 22:01:49.882588 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.882570    7082 scope.go:122] "RemoveContainer" containerID="ba50b1f9d11ae1f1988c187c47c1dbb3e6fc705c06d2009883df074a4c324d48"
Oct 29 22:01:49.882910 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:01:49.882886    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba50b1f9d11ae1f1988c187c47c1dbb3e6fc705c06d2009883df074a4c324d48\": container with ID starting with ba50b1f9d11ae1f1988c187c47c1dbb3e6fc705c06d2009883df074a4c324d48 not found: ID does not exist" containerID="ba50b1f9d11ae1f1988c187c47c1dbb3e6fc705c06d2009883df074a4c324d48"
Oct 29 22:01:49.882979 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.882922    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba50b1f9d11ae1f1988c187c47c1dbb3e6fc705c06d2009883df074a4c324d48"} err="failed to get container status \"ba50b1f9d11ae1f1988c187c47c1dbb3e6fc705c06d2009883df074a4c324d48\": rpc error: code = NotFound desc = could not find container \"ba50b1f9d11ae1f1988c187c47c1dbb3e6fc705c06d2009883df074a4c324d48\": container with ID starting with ba50b1f9d11ae1f1988c187c47c1dbb3e6fc705c06d2009883df074a4c324d48 not found: ID does not exist"
Oct 29 22:01:49.924360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.924332    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"]
Oct 29 22:01:49.928857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:49.928835    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-bhd4j"]
Oct 29 22:01:51.636918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:51.636876    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1644588d-6cb4-4f39-ab4a-04e939a89fe6" path="/var/lib/kubelet/pods/1644588d-6cb4-4f39-ab4a-04e939a89fe6/volumes"
Oct 29 22:01:55.842589 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:55.842551    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"]
Oct 29 22:01:55.843667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:55.843603    7082 cpu_manager.go:414] "containerMap: removing container" podUID="1644588d-6cb4-4f39-ab4a-04e939a89fe6" containerName="istio-proxy"
Oct 29 22:01:55.843667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:55.843655    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1644588d-6cb4-4f39-ab4a-04e939a89fe6" containerName="istio-proxy"
Oct 29 22:01:55.843860 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:55.843683    7082 cpu_manager.go:414] "containerMap: removing container" podUID="eb3e1fca-d76f-4fba-9a59-2cb8600aefda" containerName="revoke-keys"
Oct 29 22:01:55.843860 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:55.843691    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb3e1fca-d76f-4fba-9a59-2cb8600aefda" containerName="revoke-keys"
Oct 29 22:01:55.843980 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:55.843938    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="1644588d-6cb4-4f39-ab4a-04e939a89fe6" containerName="istio-proxy"
Oct 29 22:01:55.843980 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:55.843958    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb3e1fca-d76f-4fba-9a59-2cb8600aefda" containerName="revoke-keys"
Oct 29 22:01:55.848126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:55.848101    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:55.865829 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:55.865801    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"]
Oct 29 22:01:55.871744 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:55.871717    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-751a7416-openshift-default-dockercfg-sx494\""
Oct 29 22:01:56.062709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.023560    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.062709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.023619    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.062709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.023679    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.062709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.023757    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.062709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.023840    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.062709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.023878    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.062709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.023920    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.062709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.023957    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzqn\" (UniqueName: \"kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-kube-api-access-dvzqn\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.062709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.024015    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.136700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.127793    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.136700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.127858    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.136700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.127909    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.136700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.128008    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.136700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.128121    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.136700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.128165    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.136700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.128211    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.136700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.128248    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzqn\" (UniqueName: \"kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-kube-api-access-dvzqn\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.136700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.128316    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.136700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.128818    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.136700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.129475    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.136700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.131309    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.149655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.142952    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.149923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.144214    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.150024 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.144445    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.173846 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.173810    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.188038 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.188008    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.207413 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.207383    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzqn\" (UniqueName: \"kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-kube-api-access-dvzqn\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-4trvs\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.468098 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.467994    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:56.813104 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.813079    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"]
Oct 29 22:01:56.813830 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:01:56.813800    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1d3eec_cb23_48d7_bc11_bbe55580eee2.slice/crio-fb88f5c6b39d9f34c16efecb401e32686b1b1a4f84ecb9a2e102380bd8065180 WatchSource:0}: Error finding container fb88f5c6b39d9f34c16efecb401e32686b1b1a4f84ecb9a2e102380bd8065180: Status 404 returned error can't find the container with id fb88f5c6b39d9f34c16efecb401e32686b1b1a4f84ecb9a2e102380bd8065180
Oct 29 22:01:56.816689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.816652    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:01:56.816797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.816730    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:01:56.816797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:56.816779    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:01:57.011679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:57.008112    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" event={"ID":"4d1d3eec-cb23-48d7-bc11-bbe55580eee2","Type":"ContainerStarted","Data":"b2562dad0650731c90a8b6a67a45a6e3a7b099cda3ab3b46cef1356296af816c"}
Oct 29 22:01:57.011679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:57.008178    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" event={"ID":"4d1d3eec-cb23-48d7-bc11-bbe55580eee2","Type":"ContainerStarted","Data":"fb88f5c6b39d9f34c16efecb401e32686b1b1a4f84ecb9a2e102380bd8065180"}
Oct 29 22:01:57.056993 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:57.056948    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" podStartSLOduration=2.05693486 podStartE2EDuration="2.05693486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:57.053622478 +0000 UTC m=+889.553664777" watchObservedRunningTime="2026-10-29 22:01:57.05693486 +0000 UTC m=+889.556977159"
Oct 29 22:01:57.475658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:57.473822    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:57.475658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:57.473983    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:01:57.475658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:57.474071    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:01:58.415162 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:01:58.415116    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:01:58.474266 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:58.474238    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:59.028768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:59.028732    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:59.029945 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:59.029921    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:01:59.050865 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:59.050833    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"]
Oct 29 22:01:59.062351 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:59.062326    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36049559-wll2s"]
Oct 29 22:01:59.640955 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:01:59.640921    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="559e069d-73a9-4764-96ec-a1d3497def06" path="/var/lib/kubelet/pods/559e069d-73a9-4764-96ec-a1d3497def06/volumes"
Oct 29 22:02:00.030680 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.030582    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"]
Oct 29 22:02:00.035418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.035390    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"
Oct 29 22:02:00.052039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.052007    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:02:00.057855 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.057817    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"]
Oct 29 22:02:00.062385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.062340    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:02:00.062577 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.062456    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:02:00.063559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.063536    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:02:00.103363 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.103320    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aa11000-z8fmp\" (UID: \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"
Oct 29 22:02:00.103542 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.103500    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6f6b\" (UniqueName: \"kubernetes.io/projected/fd54e9e8-ebc1-4068-b199-9d0b68200da4-kube-api-access-b6f6b\") pod \"maas-api-revoke-sub-9aa11000-z8fmp\" (UID: \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"
Oct 29 22:02:00.103584 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.103563    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aa11000-z8fmp\" (UID: \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"
Oct 29 22:02:00.204815 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.204778    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aa11000-z8fmp\" (UID: \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"
Oct 29 22:02:00.205024 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.204997    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6f6b\" (UniqueName: \"kubernetes.io/projected/fd54e9e8-ebc1-4068-b199-9d0b68200da4-kube-api-access-b6f6b\") pod \"maas-api-revoke-sub-9aa11000-z8fmp\" (UID: \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"
Oct 29 22:02:00.205092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.205054    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aa11000-z8fmp\" (UID: \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"
Oct 29 22:02:00.205894 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.205871    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aa11000-z8fmp\" (UID: \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"
Oct 29 22:02:00.207385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.207365    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aa11000-z8fmp\" (UID: \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"
Oct 29 22:02:00.229981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.229944    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6f6b\" (UniqueName: \"kubernetes.io/projected/fd54e9e8-ebc1-4068-b199-9d0b68200da4-kube-api-access-b6f6b\") pod \"maas-api-revoke-sub-9aa11000-z8fmp\" (UID: \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"
Oct 29 22:02:00.352872 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.352818    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"
Oct 29 22:02:00.668252 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:02:00.668219    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd54e9e8_ebc1_4068_b199_9d0b68200da4.slice/crio-571b3c37ef8e478e9da298c1b7c0c2f590f994af70eccd487bec0fa059c2cce4 WatchSource:0}: Error finding container 571b3c37ef8e478e9da298c1b7c0c2f590f994af70eccd487bec0fa059c2cce4: Status 404 returned error can't find the container with id 571b3c37ef8e478e9da298c1b7c0c2f590f994af70eccd487bec0fa059c2cce4
Oct 29 22:02:00.668715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:00.668250    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"]
Oct 29 22:02:01.106654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:01.103417    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp" event={"ID":"fd54e9e8-ebc1-4068-b199-9d0b68200da4","Type":"ContainerStarted","Data":"571b3c37ef8e478e9da298c1b7c0c2f590f994af70eccd487bec0fa059c2cce4"}
Oct 29 22:02:01.863397 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:01.863359    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"]
Oct 29 22:02:01.903564 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:01.901796    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"]
Oct 29 22:02:01.903564 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:01.901983    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"
Oct 29 22:02:01.935509 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:01.935451    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-751a7416\""
Oct 29 22:02:01.935765 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:01.935577    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-751a7416-dockercfg-qjfb8\""
Oct 29 22:02:01.997812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:01.997783    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8da301-9bbf-4985-b81e-02e1360ab153-plugins-config-volume\") pod \"payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr\" (UID: \"fc8da301-9bbf-4985-b81e-02e1360ab153\") " pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"
Oct 29 22:02:01.998038 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:01.997821    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txd2v\" (UniqueName: \"kubernetes.io/projected/fc8da301-9bbf-4985-b81e-02e1360ab153-kube-api-access-txd2v\") pod \"payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr\" (UID: \"fc8da301-9bbf-4985-b81e-02e1360ab153\") " pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"
Oct 29 22:02:02.099726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:02.099693    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8da301-9bbf-4985-b81e-02e1360ab153-plugins-config-volume\") pod \"payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr\" (UID: \"fc8da301-9bbf-4985-b81e-02e1360ab153\") " pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"
Oct 29 22:02:02.099895 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:02.099734    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txd2v\" (UniqueName: \"kubernetes.io/projected/fc8da301-9bbf-4985-b81e-02e1360ab153-kube-api-access-txd2v\") pod \"payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr\" (UID: \"fc8da301-9bbf-4985-b81e-02e1360ab153\") " pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"
Oct 29 22:02:02.100498 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:02.100469    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8da301-9bbf-4985-b81e-02e1360ab153-plugins-config-volume\") pod \"payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr\" (UID: \"fc8da301-9bbf-4985-b81e-02e1360ab153\") " pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"
Oct 29 22:02:02.113173 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:02.113134    7082 generic.go:356] "Generic (PLEG): container finished" podID="fd54e9e8-ebc1-4068-b199-9d0b68200da4" containerID="0d6511bba247899bf06112c391bf506fe850700a1242bdbc63fd679647b4cd67" exitCode=0
Oct 29 22:02:02.113446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:02.113388    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp" event={"ID":"fd54e9e8-ebc1-4068-b199-9d0b68200da4","Type":"ContainerDied","Data":"0d6511bba247899bf06112c391bf506fe850700a1242bdbc63fd679647b4cd67"}
Oct 29 22:02:02.123163 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:02.123140    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txd2v\" (UniqueName: \"kubernetes.io/projected/fc8da301-9bbf-4985-b81e-02e1360ab153-kube-api-access-txd2v\") pod \"payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr\" (UID: \"fc8da301-9bbf-4985-b81e-02e1360ab153\") " pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"
Oct 29 22:02:02.284145 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:02.284105    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"
Oct 29 22:02:02.603088 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:02.603060    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"]
Oct 29 22:02:02.604108 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:02:02.604079    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8da301_9bbf_4985_b81e_02e1360ab153.slice/crio-b4a2a3846b298f2aca9a4172f80912c3013080d1b72b6c9c4a3978301945db0f WatchSource:0}: Error finding container b4a2a3846b298f2aca9a4172f80912c3013080d1b72b6c9c4a3978301945db0f: Status 404 returned error can't find the container with id b4a2a3846b298f2aca9a4172f80912c3013080d1b72b6c9c4a3978301945db0f
Oct 29 22:02:03.124605 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:03.124562    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr" event={"ID":"fc8da301-9bbf-4985-b81e-02e1360ab153","Type":"ContainerStarted","Data":"819862e21ce4ee98f8d718fb083ade3524862f24463e28a134696d77aa7c6607"}
Oct 29 22:02:03.125151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:03.124649    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr" event={"ID":"fc8da301-9bbf-4985-b81e-02e1360ab153","Type":"ContainerStarted","Data":"b4a2a3846b298f2aca9a4172f80912c3013080d1b72b6c9c4a3978301945db0f"}
Oct 29 22:02:03.171643 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:03.171567    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr" podStartSLOduration=2.171552827 podStartE2EDuration="2.171552827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:03.165896853 +0000 UTC m=+895.665939152" watchObservedRunningTime="2026-10-29 22:02:03.171552827 +0000 UTC m=+895.671595125"
Oct 29 22:02:03.317656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:03.307880    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"
Oct 29 22:02:03.434499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:03.434399    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd54e9e8-ebc1-4068-b199-9d0b68200da4-kube-api-access-b6f6b\" (UniqueName: \"kubernetes.io/projected/fd54e9e8-ebc1-4068-b199-9d0b68200da4-kube-api-access-b6f6b\") pod \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\" (UID: \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\") "
Oct 29 22:02:03.434705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:03.434508    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-openshift-ca\") pod \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\" (UID: \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\") "
Oct 29 22:02:03.434705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:03.434565    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-xks-ca\") pod \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\" (UID: \"fd54e9e8-ebc1-4068-b199-9d0b68200da4\") "
Oct 29 22:02:03.435030 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:03.435000    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-openshift-ca" pod "fd54e9e8-ebc1-4068-b199-9d0b68200da4" (UID: "fd54e9e8-ebc1-4068-b199-9d0b68200da4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:02:03.435407 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:03.435373    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:02:03.437203 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:03.437134    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-xks-ca" pod "fd54e9e8-ebc1-4068-b199-9d0b68200da4" (UID: "fd54e9e8-ebc1-4068-b199-9d0b68200da4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:02:03.437203 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:03.437170    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd54e9e8-ebc1-4068-b199-9d0b68200da4-kube-api-access-b6f6b" pod "fd54e9e8-ebc1-4068-b199-9d0b68200da4" (UID: "fd54e9e8-ebc1-4068-b199-9d0b68200da4"). InnerVolumeSpecName "kube-api-access-b6f6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:02:03.560662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:03.546743    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6f6b\" (UniqueName: \"kubernetes.io/projected/fd54e9e8-ebc1-4068-b199-9d0b68200da4-kube-api-access-b6f6b\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:02:03.560662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:03.546779    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd54e9e8-ebc1-4068-b199-9d0b68200da4-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:02:04.134357 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.134331    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"
Oct 29 22:02:04.134799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.134365    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp" event={"ID":"fd54e9e8-ebc1-4068-b199-9d0b68200da4","Type":"ContainerDied","Data":"571b3c37ef8e478e9da298c1b7c0c2f590f994af70eccd487bec0fa059c2cce4"}
Oct 29 22:02:04.134799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.134422    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="571b3c37ef8e478e9da298c1b7c0c2f590f994af70eccd487bec0fa059c2cce4"
Oct 29 22:02:04.134799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.134597    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"
Oct 29 22:02:04.274646 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.274596    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"]
Oct 29 22:02:04.275385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.275364    7082 cpu_manager.go:414] "containerMap: removing container" podUID="fd54e9e8-ebc1-4068-b199-9d0b68200da4" containerName="revoke-keys"
Oct 29 22:02:04.275385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.275386    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd54e9e8-ebc1-4068-b199-9d0b68200da4" containerName="revoke-keys"
Oct 29 22:02:04.275610 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.275596    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd54e9e8-ebc1-4068-b199-9d0b68200da4" containerName="revoke-keys"
Oct 29 22:02:04.280140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.280119    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"
Oct 29 22:02:04.294784 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.294764    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:02:04.294892 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.294859    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:02:04.306348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.306310    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:02:04.306348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.306320    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:02:04.317164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.317132    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"]
Oct 29 22:02:04.366011 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.365977    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g97b4\" (UniqueName: \"kubernetes.io/projected/bb42bae0-7a89-4627-baf2-c467041bacda-kube-api-access-g97b4\") pod \"maas-api-revoke-sub-eb7bc281-xvwgv\" (UID: \"bb42bae0-7a89-4627-baf2-c467041bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"
Oct 29 22:02:04.366180 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.366157    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb7bc281-xvwgv\" (UID: \"bb42bae0-7a89-4627-baf2-c467041bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"
Oct 29 22:02:04.366236 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.366219    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb7bc281-xvwgv\" (UID: \"bb42bae0-7a89-4627-baf2-c467041bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"
Oct 29 22:02:04.468056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.467959    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb7bc281-xvwgv\" (UID: \"bb42bae0-7a89-4627-baf2-c467041bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"
Oct 29 22:02:04.468230 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.468065    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb7bc281-xvwgv\" (UID: \"bb42bae0-7a89-4627-baf2-c467041bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"
Oct 29 22:02:04.468230 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.468198    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g97b4\" (UniqueName: \"kubernetes.io/projected/bb42bae0-7a89-4627-baf2-c467041bacda-kube-api-access-g97b4\") pod \"maas-api-revoke-sub-eb7bc281-xvwgv\" (UID: \"bb42bae0-7a89-4627-baf2-c467041bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"
Oct 29 22:02:04.468667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.468605    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb7bc281-xvwgv\" (UID: \"bb42bae0-7a89-4627-baf2-c467041bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"
Oct 29 22:02:04.470768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.470735    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb7bc281-xvwgv\" (UID: \"bb42bae0-7a89-4627-baf2-c467041bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"
Oct 29 22:02:04.495037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.495006    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g97b4\" (UniqueName: \"kubernetes.io/projected/bb42bae0-7a89-4627-baf2-c467041bacda-kube-api-access-g97b4\") pod \"maas-api-revoke-sub-eb7bc281-xvwgv\" (UID: \"bb42bae0-7a89-4627-baf2-c467041bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"
Oct 29 22:02:04.613287 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:04.597076    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"
Oct 29 22:02:05.020049 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:02:05.020004    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb42bae0_7a89_4627_baf2_c467041bacda.slice/crio-f0d4ab15276fd5198833d2ed1833f6017c616421916c7bb2e2558b653ea525ec WatchSource:0}: Error finding container f0d4ab15276fd5198833d2ed1833f6017c616421916c7bb2e2558b653ea525ec: Status 404 returned error can't find the container with id f0d4ab15276fd5198833d2ed1833f6017c616421916c7bb2e2558b653ea525ec
Oct 29 22:02:05.020416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:05.020395    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"]
Oct 29 22:02:05.144331 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:05.144293    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv" event={"ID":"bb42bae0-7a89-4627-baf2-c467041bacda","Type":"ContainerStarted","Data":"f0d4ab15276fd5198833d2ed1833f6017c616421916c7bb2e2558b653ea525ec"}
Oct 29 22:02:06.154436 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:06.154405    7082 generic.go:356] "Generic (PLEG): container finished" podID="bb42bae0-7a89-4627-baf2-c467041bacda" containerID="1b59e04effa4a26eb82a74fee0c0cbcc8564bf054116bbfa2f47c2bbb5488d58" exitCode=0
Oct 29 22:02:06.154866 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:06.154505    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv" event={"ID":"bb42bae0-7a89-4627-baf2-c467041bacda","Type":"ContainerDied","Data":"1b59e04effa4a26eb82a74fee0c0cbcc8564bf054116bbfa2f47c2bbb5488d58"}
Oct 29 22:02:07.358797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:07.358773    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"
Oct 29 22:02:07.442050 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:07.441910    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb42bae0-7a89-4627-baf2-c467041bacda-kube-api-access-g97b4\" (UniqueName: \"kubernetes.io/projected/bb42bae0-7a89-4627-baf2-c467041bacda-kube-api-access-g97b4\") pod \"bb42bae0-7a89-4627-baf2-c467041bacda\" (UID: \"bb42bae0-7a89-4627-baf2-c467041bacda\") "
Oct 29 22:02:07.442215 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:07.442049    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-openshift-ca\") pod \"bb42bae0-7a89-4627-baf2-c467041bacda\" (UID: \"bb42bae0-7a89-4627-baf2-c467041bacda\") "
Oct 29 22:02:07.442388 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:07.442360    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-xks-ca\") pod \"bb42bae0-7a89-4627-baf2-c467041bacda\" (UID: \"bb42bae0-7a89-4627-baf2-c467041bacda\") "
Oct 29 22:02:07.442516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:07.442421    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-openshift-ca" pod "bb42bae0-7a89-4627-baf2-c467041bacda" (UID: "bb42bae0-7a89-4627-baf2-c467041bacda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:02:07.443174 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:07.443151    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:02:07.450136 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:07.450109    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-xks-ca" pod "bb42bae0-7a89-4627-baf2-c467041bacda" (UID: "bb42bae0-7a89-4627-baf2-c467041bacda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:02:07.450136 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:07.450127    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb42bae0-7a89-4627-baf2-c467041bacda-kube-api-access-g97b4" pod "bb42bae0-7a89-4627-baf2-c467041bacda" (UID: "bb42bae0-7a89-4627-baf2-c467041bacda"). InnerVolumeSpecName "kube-api-access-g97b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:02:07.544578 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:07.544543    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g97b4\" (UniqueName: \"kubernetes.io/projected/bb42bae0-7a89-4627-baf2-c467041bacda-kube-api-access-g97b4\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:02:07.544578 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:07.544573    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb42bae0-7a89-4627-baf2-c467041bacda-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:02:08.174991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:08.174960    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"
Oct 29 22:02:08.174991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:08.174964    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv" event={"ID":"bb42bae0-7a89-4627-baf2-c467041bacda","Type":"ContainerDied","Data":"f0d4ab15276fd5198833d2ed1833f6017c616421916c7bb2e2558b653ea525ec"}
Oct 29 22:02:08.175258 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:08.175061    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0d4ab15276fd5198833d2ed1833f6017c616421916c7bb2e2558b653ea525ec"
Oct 29 22:02:08.275461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:08.275427    7082 scope.go:122] "RemoveContainer" containerID="e606676abf49bf687dc5e6fa2cf1621939a2624e59abb86401e32b9200445748"
Oct 29 22:02:08.287161 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:08.287138    7082 scope.go:122] "RemoveContainer" containerID="7e825d7b09cb5bd704757914b9b6d70ffff834ff393bf063150554d6139c2665"
Oct 29 22:02:08.303138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:08.303108    7082 scope.go:122] "RemoveContainer" containerID="6a9213b9e712955819793aab965dcf8c7743133ddf10d1e8f9f5934c91d51165"
Oct 29 22:02:08.316665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:08.316610    7082 scope.go:122] "RemoveContainer" containerID="0c93eca145c60f9b8958b134a04a5411e2d82b317f0ec680a2e40738647c2d6c"
Oct 29 22:02:08.327528 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:08.327506    7082 scope.go:122] "RemoveContainer" containerID="74780f655d34e98403e818b1c8cb3d486dcd9de3283c7638dc5bc0b01b3039c4"
Oct 29 22:02:10.364421 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.364376    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"]
Oct 29 22:02:10.365361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.365338    7082 cpu_manager.go:414] "containerMap: removing container" podUID="bb42bae0-7a89-4627-baf2-c467041bacda" containerName="revoke-keys"
Oct 29 22:02:10.365453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.365363    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb42bae0-7a89-4627-baf2-c467041bacda" containerName="revoke-keys"
Oct 29 22:02:10.366388 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.366362    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb42bae0-7a89-4627-baf2-c467041bacda" containerName="revoke-keys"
Oct 29 22:02:10.374117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.374081    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"
Oct 29 22:02:10.390182 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.390151    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"]
Oct 29 22:02:10.395937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.395874    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:02:10.396162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.396148    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:02:10.400851 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.400823    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:02:10.400990 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.400938    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:02:10.460579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.460537    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj8tw\" (UniqueName: \"kubernetes.io/projected/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-kube-api-access-qj8tw\") pod \"maas-api-revoke-sub-0f4cc552-t24zh\" (UID: \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"
Oct 29 22:02:10.460796 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.460661    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f4cc552-t24zh\" (UID: \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"
Oct 29 22:02:10.460945 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.460918    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f4cc552-t24zh\" (UID: \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"
Oct 29 22:02:10.562832 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.562791    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f4cc552-t24zh\" (UID: \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"
Oct 29 22:02:10.563051 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.562880    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj8tw\" (UniqueName: \"kubernetes.io/projected/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-kube-api-access-qj8tw\") pod \"maas-api-revoke-sub-0f4cc552-t24zh\" (UID: \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"
Oct 29 22:02:10.563051 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.562939    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f4cc552-t24zh\" (UID: \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"
Oct 29 22:02:10.563548 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.563520    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f4cc552-t24zh\" (UID: \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"
Oct 29 22:02:10.565741 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.565715    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f4cc552-t24zh\" (UID: \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"
Oct 29 22:02:10.586777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.586731    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj8tw\" (UniqueName: \"kubernetes.io/projected/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-kube-api-access-qj8tw\") pod \"maas-api-revoke-sub-0f4cc552-t24zh\" (UID: \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"
Oct 29 22:02:10.703146 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:10.703048    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"
Oct 29 22:02:11.019079 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:11.019046    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"]
Oct 29 22:02:11.021010 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:02:11.020983    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe98483_acff_4784_8ba6_c9b9ed7e46a3.slice/crio-f85794a247f75d2c52c84db8851ecae562ef23f10ccc3ca90fc0b7d2c294eefa WatchSource:0}: Error finding container f85794a247f75d2c52c84db8851ecae562ef23f10ccc3ca90fc0b7d2c294eefa: Status 404 returned error can't find the container with id f85794a247f75d2c52c84db8851ecae562ef23f10ccc3ca90fc0b7d2c294eefa
Oct 29 22:02:11.247376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:11.246406    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh" event={"ID":"fbe98483-acff-4784-8ba6-c9b9ed7e46a3","Type":"ContainerStarted","Data":"f85794a247f75d2c52c84db8851ecae562ef23f10ccc3ca90fc0b7d2c294eefa"}
Oct 29 22:02:12.256270 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:12.256235    7082 generic.go:356] "Generic (PLEG): container finished" podID="fbe98483-acff-4784-8ba6-c9b9ed7e46a3" containerID="f7e25d5bc307084fb76916e2f8002ae2805164f3a4a99b5cddeac79f39ba28a5" exitCode=0
Oct 29 22:02:12.256690 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:12.256322    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh" event={"ID":"fbe98483-acff-4784-8ba6-c9b9ed7e46a3","Type":"ContainerDied","Data":"f7e25d5bc307084fb76916e2f8002ae2805164f3a4a99b5cddeac79f39ba28a5"}
Oct 29 22:02:13.536829 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:13.536802    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"
Oct 29 22:02:13.632675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:13.632201    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-xks-ca\") pod \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\" (UID: \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\") "
Oct 29 22:02:13.632675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:13.632383    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-kube-api-access-qj8tw\" (UniqueName: \"kubernetes.io/projected/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-kube-api-access-qj8tw\") pod \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\" (UID: \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\") "
Oct 29 22:02:13.632675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:13.632446    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-openshift-ca\") pod \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\" (UID: \"fbe98483-acff-4784-8ba6-c9b9ed7e46a3\") "
Oct 29 22:02:13.635823 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:13.635791    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-openshift-ca" pod "fbe98483-acff-4784-8ba6-c9b9ed7e46a3" (UID: "fbe98483-acff-4784-8ba6-c9b9ed7e46a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:02:13.639209 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:13.639180    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-xks-ca" pod "fbe98483-acff-4784-8ba6-c9b9ed7e46a3" (UID: "fbe98483-acff-4784-8ba6-c9b9ed7e46a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:02:13.639437 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:13.639405    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-kube-api-access-qj8tw" pod "fbe98483-acff-4784-8ba6-c9b9ed7e46a3" (UID: "fbe98483-acff-4784-8ba6-c9b9ed7e46a3"). InnerVolumeSpecName "kube-api-access-qj8tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:02:13.737024 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:13.736976    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj8tw\" (UniqueName: \"kubernetes.io/projected/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-kube-api-access-qj8tw\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:02:13.737024 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:13.737008    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:02:13.737024 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:13.737022    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe98483-acff-4784-8ba6-c9b9ed7e46a3-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:02:14.094654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.092053    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"]
Oct 29 22:02:14.128467 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.127459    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f65659a3-zc4sm"]
Oct 29 22:02:14.313574 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.313544    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"
Oct 29 22:02:14.313574 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.313566    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh" event={"ID":"fbe98483-acff-4784-8ba6-c9b9ed7e46a3","Type":"ContainerDied","Data":"f85794a247f75d2c52c84db8851ecae562ef23f10ccc3ca90fc0b7d2c294eefa"}
Oct 29 22:02:14.313846 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.313616    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f85794a247f75d2c52c84db8851ecae562ef23f10ccc3ca90fc0b7d2c294eefa"
Oct 29 22:02:14.730286 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.730240    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"]
Oct 29 22:02:14.731223 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.731198    7082 cpu_manager.go:414] "containerMap: removing container" podUID="fbe98483-acff-4784-8ba6-c9b9ed7e46a3" containerName="revoke-keys"
Oct 29 22:02:14.731320 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.731225    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe98483-acff-4784-8ba6-c9b9ed7e46a3" containerName="revoke-keys"
Oct 29 22:02:14.731454 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.731440    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe98483-acff-4784-8ba6-c9b9ed7e46a3" containerName="revoke-keys"
Oct 29 22:02:14.735713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.735687    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"
Oct 29 22:02:14.756132 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.756097    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"]
Oct 29 22:02:14.761398 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.761350    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:02:14.761583 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.761406    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:02:14.761583 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.761515    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:02:14.764004 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.763972    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:02:14.877241 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.876901    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16f83bc8-s59h2\" (UID: \"e5333ebc-abe4-4164-a490-a52853830c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"
Oct 29 22:02:14.877241 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.877092    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16f83bc8-s59h2\" (UID: \"e5333ebc-abe4-4164-a490-a52853830c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"
Oct 29 22:02:14.877530 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.877260    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcx5h\" (UniqueName: \"kubernetes.io/projected/e5333ebc-abe4-4164-a490-a52853830c46-kube-api-access-wcx5h\") pod \"maas-api-revoke-sub-16f83bc8-s59h2\" (UID: \"e5333ebc-abe4-4164-a490-a52853830c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"
Oct 29 22:02:15.000760 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.998186    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16f83bc8-s59h2\" (UID: \"e5333ebc-abe4-4164-a490-a52853830c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"
Oct 29 22:02:15.000760 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.998306    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16f83bc8-s59h2\" (UID: \"e5333ebc-abe4-4164-a490-a52853830c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"
Oct 29 22:02:15.000760 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:14.998376    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcx5h\" (UniqueName: \"kubernetes.io/projected/e5333ebc-abe4-4164-a490-a52853830c46-kube-api-access-wcx5h\") pod \"maas-api-revoke-sub-16f83bc8-s59h2\" (UID: \"e5333ebc-abe4-4164-a490-a52853830c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"
Oct 29 22:02:15.001054 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:15.000816    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16f83bc8-s59h2\" (UID: \"e5333ebc-abe4-4164-a490-a52853830c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"
Oct 29 22:02:15.038657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:15.033773    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16f83bc8-s59h2\" (UID: \"e5333ebc-abe4-4164-a490-a52853830c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"
Oct 29 22:02:15.058227 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:15.057173    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcx5h\" (UniqueName: \"kubernetes.io/projected/e5333ebc-abe4-4164-a490-a52853830c46-kube-api-access-wcx5h\") pod \"maas-api-revoke-sub-16f83bc8-s59h2\" (UID: \"e5333ebc-abe4-4164-a490-a52853830c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"
Oct 29 22:02:15.079657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:15.075433    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"
Oct 29 22:02:15.148114 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:15.148067    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"
Oct 29 22:02:15.418692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:15.418659    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"]
Oct 29 22:02:15.418851 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:02:15.418787    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5333ebc_abe4_4164_a490_a52853830c46.slice/crio-33d656f5c57e5ae3854bc12926429c794357b5c376957d26abd6de1432d4af59 WatchSource:0}: Error finding container 33d656f5c57e5ae3854bc12926429c794357b5c376957d26abd6de1432d4af59: Status 404 returned error can't find the container with id 33d656f5c57e5ae3854bc12926429c794357b5c376957d26abd6de1432d4af59
Oct 29 22:02:15.634984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:15.634948    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b2dfa1e-1c25-45b8-8127-4c95f1657553" path="/var/lib/kubelet/pods/1b2dfa1e-1c25-45b8-8127-4c95f1657553/volumes"
Oct 29 22:02:16.334766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:16.334725    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2" event={"ID":"e5333ebc-abe4-4164-a490-a52853830c46","Type":"ContainerStarted","Data":"af5fd0b76632ae6e91dc828e1dd073d995a0a25af855b409c93e3a57c390c6ca"}
Oct 29 22:02:16.335245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:16.334786    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2" event={"ID":"e5333ebc-abe4-4164-a490-a52853830c46","Type":"ContainerStarted","Data":"33d656f5c57e5ae3854bc12926429c794357b5c376957d26abd6de1432d4af59"}
Oct 29 22:02:16.394618 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:16.394563    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2" podStartSLOduration=2.39454792 podStartE2EDuration="2.39454792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:16.390526101 +0000 UTC m=+908.890568393" watchObservedRunningTime="2026-10-29 22:02:16.39454792 +0000 UTC m=+908.894590237"
Oct 29 22:02:17.346557 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:17.346471    7082 generic.go:356] "Generic (PLEG): container finished" podID="e5333ebc-abe4-4164-a490-a52853830c46" containerID="af5fd0b76632ae6e91dc828e1dd073d995a0a25af855b409c93e3a57c390c6ca" exitCode=0
Oct 29 22:02:17.346945 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:17.346550    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2" event={"ID":"e5333ebc-abe4-4164-a490-a52853830c46","Type":"ContainerDied","Data":"af5fd0b76632ae6e91dc828e1dd073d995a0a25af855b409c93e3a57c390c6ca"}
Oct 29 22:02:18.555794 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:18.555763    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"
Oct 29 22:02:18.672119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:18.672030    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-xks-ca\") pod \"e5333ebc-abe4-4164-a490-a52853830c46\" (UID: \"e5333ebc-abe4-4164-a490-a52853830c46\") "
Oct 29 22:02:18.672261 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:18.672128    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5333ebc-abe4-4164-a490-a52853830c46-kube-api-access-wcx5h\" (UniqueName: \"kubernetes.io/projected/e5333ebc-abe4-4164-a490-a52853830c46-kube-api-access-wcx5h\") pod \"e5333ebc-abe4-4164-a490-a52853830c46\" (UID: \"e5333ebc-abe4-4164-a490-a52853830c46\") "
Oct 29 22:02:18.672391 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:18.672375    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-openshift-ca\") pod \"e5333ebc-abe4-4164-a490-a52853830c46\" (UID: \"e5333ebc-abe4-4164-a490-a52853830c46\") "
Oct 29 22:02:18.672820 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:18.672744    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-openshift-ca" pod "e5333ebc-abe4-4164-a490-a52853830c46" (UID: "e5333ebc-abe4-4164-a490-a52853830c46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:02:18.673173 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:18.673154    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:02:18.674492 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:18.674466    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5333ebc-abe4-4164-a490-a52853830c46-kube-api-access-wcx5h" pod "e5333ebc-abe4-4164-a490-a52853830c46" (UID: "e5333ebc-abe4-4164-a490-a52853830c46"). InnerVolumeSpecName "kube-api-access-wcx5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:02:18.674579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:18.674500    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-xks-ca" pod "e5333ebc-abe4-4164-a490-a52853830c46" (UID: "e5333ebc-abe4-4164-a490-a52853830c46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:02:18.801664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:18.800861    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5333ebc-abe4-4164-a490-a52853830c46-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:02:18.801664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:18.800897    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcx5h\" (UniqueName: \"kubernetes.io/projected/e5333ebc-abe4-4164-a490-a52853830c46-kube-api-access-wcx5h\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:02:19.365898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:19.365862    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"
Oct 29 22:02:19.366081 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:19.365892    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2" event={"ID":"e5333ebc-abe4-4164-a490-a52853830c46","Type":"ContainerDied","Data":"33d656f5c57e5ae3854bc12926429c794357b5c376957d26abd6de1432d4af59"}
Oct 29 22:02:19.366081 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:19.365946    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33d656f5c57e5ae3854bc12926429c794357b5c376957d26abd6de1432d4af59"
Oct 29 22:02:22.858726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.858678    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"]
Oct 29 22:02:22.859833 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.859806    7082 cpu_manager.go:414] "containerMap: removing container" podUID="e5333ebc-abe4-4164-a490-a52853830c46" containerName="revoke-keys"
Oct 29 22:02:22.859833 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.859833    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5333ebc-abe4-4164-a490-a52853830c46" containerName="revoke-keys"
Oct 29 22:02:22.860109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.860092    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5333ebc-abe4-4164-a490-a52853830c46" containerName="revoke-keys"
Oct 29 22:02:22.865092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.865055    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:22.889128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.889084    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"]
Oct 29 22:02:22.894416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.893346    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a9f9aa-openshift-default-dockercfg-cfh7s\""
Oct 29 22:02:22.987208 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.986999    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djdhd\" (UniqueName: \"kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-kube-api-access-djdhd\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:22.987208 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.987060    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-socket\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:22.987208 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.987154    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-envoy\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:22.987208 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.987187    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-data\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:22.987576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.987240    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istiod-ca-cert\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:22.987576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.987281    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-podinfo\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:22.987576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.987395    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-credential-socket\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:22.987576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.987571    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-token\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.214714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:22.987619    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-certs\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.340654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.339966    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-credential-socket\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.340654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.340159    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-token\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.340654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.340198    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-certs\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.340654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.340345    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdhd\" (UniqueName: \"kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-kube-api-access-djdhd\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.340654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.340381    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-socket\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.340654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.340468    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-envoy\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.340654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.340490    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-data\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.340654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.340540    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istiod-ca-cert\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.340654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.340580    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-podinfo\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.343366 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.343310    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-credential-socket\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.344180 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.344141    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-certs\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.345363 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.345319    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-data\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.345995 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.345947    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-socket\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.346910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.346866    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istiod-ca-cert\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.513598 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.505832    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-envoy\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.513598 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.507329    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-podinfo\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.513598 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.511043    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdhd\" (UniqueName: \"kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-kube-api-access-djdhd\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.513598 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.511372    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-token\") pod \"e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") " pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:23.780868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:23.780786    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:24.161660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:24.155350    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"]
Oct 29 22:02:24.248127 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:24.248077    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:02:24.248375 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:24.248353    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:02:24.248499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:24.248484    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:02:24.517380 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:24.517248    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg" event={"ID":"9f82ffb2-04a4-4fff-916f-2b06335dd7bf","Type":"ContainerStarted","Data":"9f30503010576ed612a3ef94b254ddc93968b5efb94f25f50e2da8b6669115e5"}
Oct 29 22:02:24.517380 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:24.517314    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg" event={"ID":"9f82ffb2-04a4-4fff-916f-2b06335dd7bf","Type":"ContainerStarted","Data":"ecddc087ef42b8efb5b1d1983bf693449e6df606a05797da9f7686aceaf6e5cb"}
Oct 29 22:02:24.574996 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:24.574934    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg" podStartSLOduration=2.57491492 podStartE2EDuration="2.57491492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:24.566315433 +0000 UTC m=+917.066357733" watchObservedRunningTime="2026-10-29 22:02:24.57491492 +0000 UTC m=+917.074957274"
Oct 29 22:02:24.784717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:24.781918    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:25.838702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:25.838359    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:26.038936 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:26.038900    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"]
Oct 29 22:02:26.043436 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:26.043412    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5055b163-twtsr"]
Oct 29 22:02:26.538722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:26.538689    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:26.539604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:26.539587    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:02:27.635619 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:27.635579    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4d4672-2bd0-41fb-8bff-f0cfb2469e08" path="/var/lib/kubelet/pods/5e4d4672-2bd0-41fb-8bff-f0cfb2469e08/volumes"
Oct 29 22:02:28.417342 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:02:28.417299    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:02:45.039728 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:45.039690    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"]
Oct 29 22:02:45.050891 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:45.050865    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78774571-5q2hc"]
Oct 29 22:02:45.637030 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:45.636997    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="079f1f32-0b82-4c00-90fa-3eddd0260866" path="/var/lib/kubelet/pods/079f1f32-0b82-4c00-90fa-3eddd0260866/volumes"
Oct 29 22:02:49.158654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:49.149689    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:02:53.199653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.186950    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"]
Oct 29 22:02:53.199653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.195707    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:02:53.423653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.421282    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.439906 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.439871    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"]
Oct 29 22:02:53.440343 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.440176    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.441778 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.440276    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"
Oct 29 22:02:53.487711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.460001    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:02:53.487711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.460084    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:02:53.492880 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.492850    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:02:53.493190 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.493172    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:02:53.493733 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.493686    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.532673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.506715    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.538918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.538535    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.553271 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.553217    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d14b5e7c-p2cmw\" (UID: \"2198e677-436e-4fea-838e-cbc5881dc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"
Oct 29 22:02:53.553824 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.553785    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2f5g\" (UniqueName: \"kubernetes.io/projected/2198e677-436e-4fea-838e-cbc5881dc6e8-kube-api-access-l2f5g\") pod \"maas-api-revoke-sub-d14b5e7c-p2cmw\" (UID: \"2198e677-436e-4fea-838e-cbc5881dc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"
Oct 29 22:02:53.565457 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.565430    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d14b5e7c-p2cmw\" (UID: \"2198e677-436e-4fea-838e-cbc5881dc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"
Oct 29 22:02:53.669586 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.669543    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2f5g\" (UniqueName: \"kubernetes.io/projected/2198e677-436e-4fea-838e-cbc5881dc6e8-kube-api-access-l2f5g\") pod \"maas-api-revoke-sub-d14b5e7c-p2cmw\" (UID: \"2198e677-436e-4fea-838e-cbc5881dc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"
Oct 29 22:02:53.670260 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.670229    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d14b5e7c-p2cmw\" (UID: \"2198e677-436e-4fea-838e-cbc5881dc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"
Oct 29 22:02:53.690396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.675182    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d14b5e7c-p2cmw\" (UID: \"2198e677-436e-4fea-838e-cbc5881dc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"
Oct 29 22:02:53.690396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.687280    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d14b5e7c-p2cmw\" (UID: \"2198e677-436e-4fea-838e-cbc5881dc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"
Oct 29 22:02:53.934687 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.904585    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d14b5e7c-p2cmw\" (UID: \"2198e677-436e-4fea-838e-cbc5881dc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"
Oct 29 22:02:53.981039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:53.980996    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:02:54.112838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.098455    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.208337 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.207596    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.351930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.351899    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:02:54.444962 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.444902    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:02:54.464985 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.464897    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:02:54.520651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.500720    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"]
Oct 29 22:02:54.523363 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.522425    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:02:54.524138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.524115    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:02:54.533803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.533764    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.534597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.534573    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:02:54.628828 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.628685    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.653785 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.653744    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:02:54.711496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.692855    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f840c335-rszds"]
Oct 29 22:02:54.758776 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.747774    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2f5g\" (UniqueName: \"kubernetes.io/projected/2198e677-436e-4fea-838e-cbc5881dc6e8-kube-api-access-l2f5g\") pod \"maas-api-revoke-sub-d14b5e7c-p2cmw\" (UID: \"2198e677-436e-4fea-838e-cbc5881dc6e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"
Oct 29 22:02:54.758776 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.747875    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.820516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:54.820235    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"
Oct 29 22:02:55.071150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:55.056547    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.105653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:55.072008    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:02:55.105653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:55.101403    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": context deadline exceeded"
Oct 29 22:02:55.106753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:55.105981    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.106753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:55.106431    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:02:55.224661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:55.216803    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.517114 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:55.517067    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.544814 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:55.544767    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.623382 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:55.556419    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.646761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:55.646712    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.113768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:56.113727    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:56.129657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:56.118816    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.134879 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:56.101729    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.147371 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:56.147332    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5bbe37-aae9-4c32-96a0-dbc09a566fd6" path="/var/lib/kubelet/pods/ed5bbe37-aae9-4c32-96a0-dbc09a566fd6/volumes"
Oct 29 22:02:56.150841 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:56.148081    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"]
Oct 29 22:02:56.307524 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:56.264147    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7f7a1cf-25dlm"]
Oct 29 22:02:56.307524 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:56.217785    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.307524 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:56.120002    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.552785 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:56.552737    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.638724 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:56.637365    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:02:56.701330 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:56.637422    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.726997 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:56.688755    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.183655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:57.142025    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.267761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:57.267714    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.287058 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:57.282058    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.720670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:57.703948    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:02:57.720670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:57.705033    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.738085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:57.738035    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:02:57.869709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:57.852196    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.117016 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.116971    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded"
Oct 29 22:02:58.119560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.119526    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:58.119835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.119775    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.194649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.178887    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:58.289740 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.286836    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.289740 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.287059    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.319336 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.319289    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.355971 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.355910    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:58.380813 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.380715    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.454513 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.453789    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.704107 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.704071    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:58.717685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.704349    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.737658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.732706    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.745663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.744804    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.821863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.821823    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.861889 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.861808    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.875085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.871432    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b091f9da-2dd9-466b-b7b1-31b685b71481" path="/var/lib/kubelet/pods/b091f9da-2dd9-466b-b7b1-31b685b71481/volumes"
Oct 29 22:02:58.897654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:58.893999    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"]
Oct 29 22:02:59.299655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:59.299582    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:02:59.320007 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:59.319961    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.507396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:59.507309    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded"
Oct 29 22:02:59.529098 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:59.524164    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:02:59.623062 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:02:59.623010    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:02:59.624123 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:59.624077    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw" event={"ID":"2198e677-436e-4fea-838e-cbc5881dc6e8","Type":"ContainerStarted","Data":"423ccd23f05e2c1313c369d3dc179840d00dfcf896d8f255381f30a302e1054d"}
Oct 29 22:02:59.879783 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:59.879730    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.880845 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:59.880408    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.881841 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:59.880439    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded"
Oct 29 22:02:59.882515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:59.880461    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:02:59.886728 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:02:59.886697    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.176859 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:00.176820    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:00.177183 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:00.177135    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.308948 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:00.308867    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:03:00.321785 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:00.320713    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.581409 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:00.580874    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.685654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:00.674883    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw" event={"ID":"2198e677-436e-4fea-838e-cbc5881dc6e8","Type":"ContainerStarted","Data":"49f86a0498b29ba75b734a43a50a0c3244e9389a11448f3bd1b17cc72a842235"}
Oct 29 22:03:00.760653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:00.759693    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw" podStartSLOduration=8.759670589 podStartE2EDuration="8.759670589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:00.724450807 +0000 UTC m=+953.224493106" watchObservedRunningTime="2026-10-29 22:03:00.759670589 +0000 UTC m=+953.259712889"
Oct 29 22:03:00.883706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:00.880616    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:03:00.885828 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:00.884203    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.885828 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:00.885080    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.894458 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:00.892751    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.008448 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:01.006759    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:01.008448 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:01.006825    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.335653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:01.326844    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.335653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:01.328131    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.614733 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:01.599767    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.930139 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:01.912140    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.930139 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:01.912488    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.930139 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:01.912773    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:03:01.930139 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:01.913027    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.175797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:02.175745    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.228860 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:02.228772    7082 generic.go:356] "Generic (PLEG): container finished" podID="2198e677-436e-4fea-838e-cbc5881dc6e8" containerID="49f86a0498b29ba75b734a43a50a0c3244e9389a11448f3bd1b17cc72a842235" exitCode=0
Oct 29 22:03:02.229316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:02.229286    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw" event={"ID":"2198e677-436e-4fea-838e-cbc5881dc6e8","Type":"ContainerDied","Data":"49f86a0498b29ba75b734a43a50a0c3244e9389a11448f3bd1b17cc72a842235"}
Oct 29 22:03:02.328358 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:02.328307    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.328934 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:02.328901    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:03:02.721670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:02.710016    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.959029 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:02.946646    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:03:02.959029 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:02.947081    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:03:02.959029 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:02.947424    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:03:02.959029 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:02.947776    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:03:03.362650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:03.358788    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:03:03.362650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:03.362087    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.751853 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:03.750109    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:03:03.875168 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:03.856955    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.928347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:03.928308    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:03.928670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:03.928614    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.972912 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:03.962743    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:03:03.972912 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:03.963113    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:03:03.972912 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:03.963347    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:03:03.972912 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:03.963557    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:03:04.022530 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.022439    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"]
Oct 29 22:03:04.405576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.405518    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:04.444780 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.359578    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:03:04.713848 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.711248    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:04.713848 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.711336    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:04.741654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.736867    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:04.741654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.736950    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:04.745875 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.745840    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"]
Oct 29 22:03:04.767727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.747237    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"
Oct 29 22:03:04.768594 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.758730    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:04.804108 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.804069    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:04.804411 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.804365    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:04.822109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.822058    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:04.942656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.941927    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9kmf\" (UniqueName: \"kubernetes.io/projected/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-kube-api-access-h9kmf\") pod \"maas-api-revoke-sub-1dfe75a0-n4l5n\" (UID: \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"
Oct 29 22:03:04.942656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.942069    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dfe75a0-n4l5n\" (UID: \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"
Oct 29 22:03:04.942656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.942409    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dfe75a0-n4l5n\" (UID: \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"
Oct 29 22:03:04.975848 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.975734    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:05.005648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.978894    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:03:05.005935 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.978929    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:03:05.006077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:04.978948    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:03:05.044665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:05.044596    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:05.067432 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:05.046365    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9kmf\" (UniqueName: \"kubernetes.io/projected/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-kube-api-access-h9kmf\") pod \"maas-api-revoke-sub-1dfe75a0-n4l5n\" (UID: \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"
Oct 29 22:03:05.076847 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:05.070973    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dfe75a0-n4l5n\" (UID: \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"
Oct 29 22:03:05.077376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:05.077349    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dfe75a0-n4l5n\" (UID: \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"
Oct 29 22:03:05.078175 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:05.072825    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dfe75a0-n4l5n\" (UID: \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"
Oct 29 22:03:05.151721 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:05.151687    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dfe75a0-n4l5n\" (UID: \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"
Oct 29 22:03:05.173930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:05.173836    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9kmf\" (UniqueName: \"kubernetes.io/projected/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-kube-api-access-h9kmf\") pod \"maas-api-revoke-sub-1dfe75a0-n4l5n\" (UID: \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"
Oct 29 22:03:05.268718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:05.268608    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"
Oct 29 22:03:05.459659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:05.439567    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:05.463681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:05.449762    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:05.793279 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:05.792565    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:03:06.021672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.011785    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:03:06.021672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.012278    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:06.022329 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.022297    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:03:06.034650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.022733    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:06.410895 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.410788    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"
Oct 29 22:03:06.510658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.503918    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:03:06.572661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.524484    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-xks-ca\") pod \"2198e677-436e-4fea-838e-cbc5881dc6e8\" (UID: \"2198e677-436e-4fea-838e-cbc5881dc6e8\") "
Oct 29 22:03:06.572661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.524595    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2198e677-436e-4fea-838e-cbc5881dc6e8-kube-api-access-l2f5g\" (UniqueName: \"kubernetes.io/projected/2198e677-436e-4fea-838e-cbc5881dc6e8-kube-api-access-l2f5g\") pod \"2198e677-436e-4fea-838e-cbc5881dc6e8\" (UID: \"2198e677-436e-4fea-838e-cbc5881dc6e8\") "
Oct 29 22:03:06.572661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.524724    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-openshift-ca\") pod \"2198e677-436e-4fea-838e-cbc5881dc6e8\" (UID: \"2198e677-436e-4fea-838e-cbc5881dc6e8\") "
Oct 29 22:03:06.581267 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.580759    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:06.618054 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.618012    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-openshift-ca" pod "2198e677-436e-4fea-838e-cbc5881dc6e8" (UID: "2198e677-436e-4fea-838e-cbc5881dc6e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:03:06.674221 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.673984    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2198e677-436e-4fea-838e-cbc5881dc6e8-kube-api-access-l2f5g" pod "2198e677-436e-4fea-838e-cbc5881dc6e8" (UID: "2198e677-436e-4fea-838e-cbc5881dc6e8"). InnerVolumeSpecName "kube-api-access-l2f5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:03:06.742852 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.742018    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-xks-ca" pod "2198e677-436e-4fea-838e-cbc5881dc6e8" (UID: "2198e677-436e-4fea-838e-cbc5881dc6e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:03:06.761686 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.744563    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:06.761686 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.744599    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2f5g\" (UniqueName: \"kubernetes.io/projected/2198e677-436e-4fea-838e-cbc5881dc6e8-kube-api-access-l2f5g\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:06.761686 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.744615    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2198e677-436e-4fea-838e-cbc5881dc6e8-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:06.793387 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.792758    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:03:06.816494 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:06.816460    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:03:07.067092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.046144    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:07.112120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.086577    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:07.112120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.086780    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:07.442491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.430258    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw" event={"ID":"2198e677-436e-4fea-838e-cbc5881dc6e8","Type":"ContainerDied","Data":"423ccd23f05e2c1313c369d3dc179840d00dfcf896d8f255381f30a302e1054d"}
Oct 29 22:03:07.442491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.430321    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="423ccd23f05e2c1313c369d3dc179840d00dfcf896d8f255381f30a302e1054d"
Oct 29 22:03:07.442491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.430398    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"
Oct 29 22:03:07.479685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.474964    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:03:07.528761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.528715    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:07.593996 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.591940    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:07.623331 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.619175    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:03:07.949521 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.944028    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:03:07.949521 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.944171    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:03:07.949521 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.944191    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"]
Oct 29 22:03:07.949521 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.944241    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:03:07.949521 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:07.944271    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:03:08.412902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:08.406390    7082 scope.go:122] "RemoveContainer" containerID="e5acced70ba8d2bdbaaab9508de630f984d8692e62e5cfb919baaf93571454ee"
Oct 29 22:03:08.464697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:08.464586    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n" event={"ID":"b937919a-4a86-4a6a-be3e-a0c9a105d0f7","Type":"ContainerStarted","Data":"9be9c47dd01c19614433592fa13fdc1c0587cda6262e3069a1ddf6b6e434a0eb"}
Oct 29 22:03:09.478642 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:09.478590    7082 generic.go:356] "Generic (PLEG): container finished" podID="b937919a-4a86-4a6a-be3e-a0c9a105d0f7" containerID="2e9291d2ae8808796ad29701fa4e2966804d433c272a19f51aaa05ddc2a5f231" exitCode=0
Oct 29 22:03:09.479375 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:09.478673    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n" event={"ID":"b937919a-4a86-4a6a-be3e-a0c9a105d0f7","Type":"ContainerDied","Data":"2e9291d2ae8808796ad29701fa4e2966804d433c272a19f51aaa05ddc2a5f231"}
Oct 29 22:03:10.764854 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:10.764831    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"
Oct 29 22:03:10.955592 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:10.952765    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-kube-api-access-h9kmf\" (UniqueName: \"kubernetes.io/projected/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-kube-api-access-h9kmf\") pod \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\" (UID: \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\") "
Oct 29 22:03:10.955592 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:10.952910    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-openshift-ca\") pod \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\" (UID: \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\") "
Oct 29 22:03:10.971191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:10.971150    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-openshift-ca" pod "b937919a-4a86-4a6a-be3e-a0c9a105d0f7" (UID: "b937919a-4a86-4a6a-be3e-a0c9a105d0f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:03:10.975232 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:10.975185    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-xks-ca\") pod \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\" (UID: \"b937919a-4a86-4a6a-be3e-a0c9a105d0f7\") "
Oct 29 22:03:10.978811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:10.978780    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:11.016918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.011233    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-kube-api-access-h9kmf" pod "b937919a-4a86-4a6a-be3e-a0c9a105d0f7" (UID: "b937919a-4a86-4a6a-be3e-a0c9a105d0f7"). InnerVolumeSpecName "kube-api-access-h9kmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:03:11.017429 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.017392    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-xks-ca" pod "b937919a-4a86-4a6a-be3e-a0c9a105d0f7" (UID: "b937919a-4a86-4a6a-be3e-a0c9a105d0f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:03:11.098663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.097239    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:11.098663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.097274    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9kmf\" (UniqueName: \"kubernetes.io/projected/b937919a-4a86-4a6a-be3e-a0c9a105d0f7-kube-api-access-h9kmf\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:11.214046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.213954    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"]
Oct 29 22:03:11.214731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.214711    7082 cpu_manager.go:414] "containerMap: removing container" podUID="b937919a-4a86-4a6a-be3e-a0c9a105d0f7" containerName="revoke-keys"
Oct 29 22:03:11.214731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.214726    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b937919a-4a86-4a6a-be3e-a0c9a105d0f7" containerName="revoke-keys"
Oct 29 22:03:11.214929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.214747    7082 cpu_manager.go:414] "containerMap: removing container" podUID="2198e677-436e-4fea-838e-cbc5881dc6e8" containerName="revoke-keys"
Oct 29 22:03:11.214929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.214752    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2198e677-436e-4fea-838e-cbc5881dc6e8" containerName="revoke-keys"
Oct 29 22:03:11.214929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.214879    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="b937919a-4a86-4a6a-be3e-a0c9a105d0f7" containerName="revoke-keys"
Oct 29 22:03:11.214929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.214890    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="2198e677-436e-4fea-838e-cbc5881dc6e8" containerName="revoke-keys"
Oct 29 22:03:11.219092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.219070    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.252196 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.250862    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"]
Oct 29 22:03:11.300438 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.300409    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.300594 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.300530    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.300674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.300636    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.300789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.300777    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.300853 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.300839    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.300891 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.300882    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.300958 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.300946    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6jc\" (UniqueName: \"kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-kube-api-access-rk6jc\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.300996 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.300985    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.301053 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.301007    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.402306 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.402267    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.402491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.402351    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.402491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.402388    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.402617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.402538    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.402915 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.402890    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.402986 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.402898    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.402986 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.402965    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.403076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.403002    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.403076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.403070    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6jc\" (UniqueName: \"kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-kube-api-access-rk6jc\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.403188 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.403132    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.403188 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.403165    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.403289 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.403221    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.403567 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.403545    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.403937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.403914    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.405583 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.405558    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.405692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.405620    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.426037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.426015    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.426150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.426132    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6jc\" (UniqueName: \"kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-kube-api-access-rk6jc\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.510617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.510530    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n" event={"ID":"b937919a-4a86-4a6a-be3e-a0c9a105d0f7","Type":"ContainerDied","Data":"9be9c47dd01c19614433592fa13fdc1c0587cda6262e3069a1ddf6b6e434a0eb"}
Oct 29 22:03:11.510617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.510572    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"
Oct 29 22:03:11.510617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.510593    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9be9c47dd01c19614433592fa13fdc1c0587cda6262e3069a1ddf6b6e434a0eb"
Oct 29 22:03:11.554854 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.554828    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:11.908566 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:03:11.908523    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b822e3a_e780_4b65_98bc_f0c26990bfcf.slice/crio-ee2b5be7120a269def776611ce4ebf0d6c618f329c83b31c4732b8d0c3663ed0 WatchSource:0}: Error finding container ee2b5be7120a269def776611ce4ebf0d6c618f329c83b31c4732b8d0c3663ed0: Status 404 returned error can't find the container with id ee2b5be7120a269def776611ce4ebf0d6c618f329c83b31c4732b8d0c3663ed0
Oct 29 22:03:11.911412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.911375    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"]
Oct 29 22:03:11.912035 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.911995    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:03:11.912200 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.912082    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:03:11.912200 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:11.912128    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:03:12.527054 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:12.526982    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" event={"ID":"6b822e3a-e780-4b65-98bc-f0c26990bfcf","Type":"ContainerStarted","Data":"800159fb2e9afbc67db3c9c6b50fec50328770c8edbf2efd22113a417c37e2df"}
Oct 29 22:03:12.527246 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:12.527085    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" event={"ID":"6b822e3a-e780-4b65-98bc-f0c26990bfcf","Type":"ContainerStarted","Data":"ee2b5be7120a269def776611ce4ebf0d6c618f329c83b31c4732b8d0c3663ed0"}
Oct 29 22:03:12.555951 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:12.555916    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:12.557748 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:12.557713    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:03:12.557916 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:12.557769    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" podUID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:03:12.576415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:12.576358    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" podStartSLOduration=1.576341545 podStartE2EDuration="1.576341545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:12.574924768 +0000 UTC m=+965.074967068" watchObservedRunningTime="2026-10-29 22:03:12.576341545 +0000 UTC m=+965.076383844"
Oct 29 22:03:13.555904 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:13.555857    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:03:13.556406 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:13.555927    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" podUID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:03:14.567663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:14.564716    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:03:14.567663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:14.564780    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" podUID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:03:15.100295 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:15.100255    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:15.100643 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:15.100592    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:15.556163 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:15.556131    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:03:15.556316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:15.556184    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" podUID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:03:16.265482 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.265436    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"]
Oct 29 22:03:16.333698 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.319536    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"]
Oct 29 22:03:16.333698 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.319685    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"
Oct 29 22:03:16.336819 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.336700    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:03:16.336819 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.336757    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:03:16.336819 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.336781    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:03:16.337089 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.336900    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:03:16.376540 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.376481    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spd82\" (UniqueName: \"kubernetes.io/projected/86a87ce0-f034-4a27-aee7-2045c8c21ab4-kube-api-access-spd82\") pod \"maas-api-revoke-sub-93a248d5-lhmll\" (UID: \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"
Oct 29 22:03:16.376540 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.376534    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a248d5-lhmll\" (UID: \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"
Oct 29 22:03:16.376787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.376663    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a248d5-lhmll\" (UID: \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"
Oct 29 22:03:16.480617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.479572    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spd82\" (UniqueName: \"kubernetes.io/projected/86a87ce0-f034-4a27-aee7-2045c8c21ab4-kube-api-access-spd82\") pod \"maas-api-revoke-sub-93a248d5-lhmll\" (UID: \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"
Oct 29 22:03:16.480617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.479652    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a248d5-lhmll\" (UID: \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"
Oct 29 22:03:16.480617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.479782    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a248d5-lhmll\" (UID: \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"
Oct 29 22:03:16.504674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.482178    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a248d5-lhmll\" (UID: \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"
Oct 29 22:03:16.530695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.514966    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a248d5-lhmll\" (UID: \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"
Oct 29 22:03:16.571242 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.564794    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:03:16.571242 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.564871    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" podUID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:03:16.583279 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.579835    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spd82\" (UniqueName: \"kubernetes.io/projected/86a87ce0-f034-4a27-aee7-2045c8c21ab4-kube-api-access-spd82\") pod \"maas-api-revoke-sub-93a248d5-lhmll\" (UID: \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"
Oct 29 22:03:16.677703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:16.677464    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"
Oct 29 22:03:17.370014 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:17.369982    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"]
Oct 29 22:03:17.370359 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:03:17.370198    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a87ce0_f034_4a27_aee7_2045c8c21ab4.slice/crio-4c2d020792b7061e2e7ad78f3da5a8814858b5ae3ddb38d25cfb143b7cd75c53 WatchSource:0}: Error finding container 4c2d020792b7061e2e7ad78f3da5a8814858b5ae3ddb38d25cfb143b7cd75c53: Status 404 returned error can't find the container with id 4c2d020792b7061e2e7ad78f3da5a8814858b5ae3ddb38d25cfb143b7cd75c53
Oct 29 22:03:17.556067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:17.556031    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:03:17.556250 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:17.556096    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" podUID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:03:17.740116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:17.740078    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll" event={"ID":"86a87ce0-f034-4a27-aee7-2045c8c21ab4","Type":"ContainerStarted","Data":"4c2d020792b7061e2e7ad78f3da5a8814858b5ae3ddb38d25cfb143b7cd75c53"}
Oct 29 22:03:18.578753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:18.561664    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:03:18.578753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:18.561730    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" podUID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:03:18.750387 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:18.750350    7082 generic.go:356] "Generic (PLEG): container finished" podID="86a87ce0-f034-4a27-aee7-2045c8c21ab4" containerID="19d07e36c3b8e9325d2ae74625023a21c6cbcb93cf7f3a8a6d12ed170a753572" exitCode=0
Oct 29 22:03:18.750620 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:18.750597    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll" event={"ID":"86a87ce0-f034-4a27-aee7-2045c8c21ab4","Type":"ContainerDied","Data":"19d07e36c3b8e9325d2ae74625023a21c6cbcb93cf7f3a8a6d12ed170a753572"}
Oct 29 22:03:19.560893 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:19.560864    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:19.561919 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:19.561287    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:19.561991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:19.561968    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:03:19.772422 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:19.772335    7082 generic.go:356] "Generic (PLEG): container finished" podID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerID="b2562dad0650731c90a8b6a67a45a6e3a7b099cda3ab3b46cef1356296af816c" exitCode=0
Oct 29 22:03:19.773592 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:19.773567    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" event={"ID":"4d1d3eec-cb23-48d7-bc11-bbe55580eee2","Type":"ContainerDied","Data":"b2562dad0650731c90a8b6a67a45a6e3a7b099cda3ab3b46cef1356296af816c"}
Oct 29 22:03:19.774056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:19.774036    7082 scope.go:122] "RemoveContainer" containerID="b2562dad0650731c90a8b6a67a45a6e3a7b099cda3ab3b46cef1356296af816c"
Oct 29 22:03:19.776739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:19.776701    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:03:19.776846 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:19.776796    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:03:19.776886 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:19.776843    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:03:20.072430 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.072402    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"
Oct 29 22:03:20.136310 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.136223    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-xks-ca\") pod \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\" (UID: \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\") "
Oct 29 22:03:20.136463 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.136358    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a87ce0-f034-4a27-aee7-2045c8c21ab4-kube-api-access-spd82\" (UniqueName: \"kubernetes.io/projected/86a87ce0-f034-4a27-aee7-2045c8c21ab4-kube-api-access-spd82\") pod \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\" (UID: \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\") "
Oct 29 22:03:20.136687 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.136665    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-openshift-ca\") pod \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\" (UID: \"86a87ce0-f034-4a27-aee7-2045c8c21ab4\") "
Oct 29 22:03:20.137258 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.137147    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-openshift-ca" pod "86a87ce0-f034-4a27-aee7-2045c8c21ab4" (UID: "86a87ce0-f034-4a27-aee7-2045c8c21ab4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:03:20.137444 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.137427    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:20.152380 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.152298    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-xks-ca" pod "86a87ce0-f034-4a27-aee7-2045c8c21ab4" (UID: "86a87ce0-f034-4a27-aee7-2045c8c21ab4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:03:20.153834 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.153810    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a87ce0-f034-4a27-aee7-2045c8c21ab4-kube-api-access-spd82" pod "86a87ce0-f034-4a27-aee7-2045c8c21ab4" (UID: "86a87ce0-f034-4a27-aee7-2045c8c21ab4"). InnerVolumeSpecName "kube-api-access-spd82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:03:20.238580 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.238547    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a87ce0-f034-4a27-aee7-2045c8c21ab4-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:20.238580 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.238578    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spd82\" (UniqueName: \"kubernetes.io/projected/86a87ce0-f034-4a27-aee7-2045c8c21ab4-kube-api-access-spd82\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:20.620143 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.620104    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"]
Oct 29 22:03:20.620990 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.620966    7082 cpu_manager.go:414] "containerMap: removing container" podUID="86a87ce0-f034-4a27-aee7-2045c8c21ab4" containerName="revoke-keys"
Oct 29 22:03:20.620990 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.620991    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a87ce0-f034-4a27-aee7-2045c8c21ab4" containerName="revoke-keys"
Oct 29 22:03:20.621186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.621170    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a87ce0-f034-4a27-aee7-2045c8c21ab4" containerName="revoke-keys"
Oct 29 22:03:20.624001 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.623970    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"
Oct 29 22:03:20.640079 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.640054    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6df539\""
Oct 29 22:03:20.644025 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.644002    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"]
Oct 29 22:03:20.775659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.748535    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7926\" (UniqueName: \"kubernetes.io/projected/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-kube-api-access-k7926\") pod \"payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz\" (UID: \"26bfc7bc-ef5e-4df2-998b-bdc774f39f89\") " pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"
Oct 29 22:03:20.860777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.833030    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz\" (UID: \"26bfc7bc-ef5e-4df2-998b-bdc774f39f89\") " pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"
Oct 29 22:03:21.030009 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.999723    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz\" (UID: \"26bfc7bc-ef5e-4df2-998b-bdc774f39f89\") " pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"
Oct 29 22:03:21.030009 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:20.999922    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7926\" (UniqueName: \"kubernetes.io/projected/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-kube-api-access-k7926\") pod \"payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz\" (UID: \"26bfc7bc-ef5e-4df2-998b-bdc774f39f89\") " pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"
Oct 29 22:03:21.030009 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:21.001140    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz\" (UID: \"26bfc7bc-ef5e-4df2-998b-bdc774f39f89\") " pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"
Oct 29 22:03:21.056512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:21.056471    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7926\" (UniqueName: \"kubernetes.io/projected/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-kube-api-access-k7926\") pod \"payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz\" (UID: \"26bfc7bc-ef5e-4df2-998b-bdc774f39f89\") " pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"
Oct 29 22:03:21.076388 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:21.076352    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" event={"ID":"4d1d3eec-cb23-48d7-bc11-bbe55580eee2","Type":"ContainerStarted","Data":"efee2de6f820641c7f87224c2c71d0a7984816e3e82ddfcec7b2ccc1c2b84800"}
Oct 29 22:03:21.127609 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:21.127578    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"
Oct 29 22:03:21.131741 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:21.131703    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll" event={"ID":"86a87ce0-f034-4a27-aee7-2045c8c21ab4","Type":"ContainerDied","Data":"4c2d020792b7061e2e7ad78f3da5a8814858b5ae3ddb38d25cfb143b7cd75c53"}
Oct 29 22:03:21.131977 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:21.131962    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c2d020792b7061e2e7ad78f3da5a8814858b5ae3ddb38d25cfb143b7cd75c53"
Oct 29 22:03:21.238741 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:21.238430    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"
Oct 29 22:03:21.379229 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:03:21.379188    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a87ce0_f034_4a27_aee7_2045c8c21ab4.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:03:21.470429 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:21.470398    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:03:21.470665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:21.470585    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:03:21.470756 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:21.470656    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:03:21.645963 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:03:21.645934    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26bfc7bc_ef5e_4df2_998b_bdc774f39f89.slice/crio-f281df2ed6fcf424dd00e007468eb2db8b91e17681d1f02ecd2871389f27c09f WatchSource:0}: Error finding container f281df2ed6fcf424dd00e007468eb2db8b91e17681d1f02ecd2871389f27c09f: Status 404 returned error can't find the container with id f281df2ed6fcf424dd00e007468eb2db8b91e17681d1f02ecd2871389f27c09f
Oct 29 22:03:21.646164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:21.646141    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"]
Oct 29 22:03:22.138273 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:22.138236    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz" event={"ID":"26bfc7bc-ef5e-4df2-998b-bdc774f39f89","Type":"ContainerStarted","Data":"62e3f8b61fe46b3b51fd2cb1a2bc7e91ab091fce77b94ea5bf9ea7cff9abb4f5"}
Oct 29 22:03:22.138704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:22.138294    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz" event={"ID":"26bfc7bc-ef5e-4df2-998b-bdc774f39f89","Type":"ContainerStarted","Data":"f281df2ed6fcf424dd00e007468eb2db8b91e17681d1f02ecd2871389f27c09f"}
Oct 29 22:03:22.184502 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:22.184453    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz" podStartSLOduration=2.184429734 podStartE2EDuration="2.184429734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:22.182934422 +0000 UTC m=+974.682976722" watchObservedRunningTime="2026-10-29 22:03:22.184429734 +0000 UTC m=+974.684472064"
Oct 29 22:03:22.491717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:22.482435    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:03:22.491717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:22.482496    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:03:23.148607 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:23.148577    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"
Oct 29 22:03:23.470118 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:23.470025    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:03:23.470118 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:23.470100    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:03:24.470518 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:24.470477    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:03:24.470946 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:24.470551    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:03:25.471059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:25.471018    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:03:25.471595 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:25.471085    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:03:26.470366 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.470326    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:03:26.470547 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.470390    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:03:26.739457 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.739356    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"]
Oct 29 22:03:26.742915 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.742892    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"
Oct 29 22:03:26.772655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.770019    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:03:26.772655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.770101    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:03:26.772655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.770138    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:03:26.772655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.770234    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:03:26.777434 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.777405    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"]
Oct 29 22:03:26.833909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.833878    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-178f1555-t8brh\" (UID: \"c13290d1-0bf7-40dc-9e78-6e450afbda03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"
Oct 29 22:03:26.834305 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.834284    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-178f1555-t8brh\" (UID: \"c13290d1-0bf7-40dc-9e78-6e450afbda03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"
Oct 29 22:03:26.834521 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.834507    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwgq\" (UniqueName: \"kubernetes.io/projected/c13290d1-0bf7-40dc-9e78-6e450afbda03-kube-api-access-2dwgq\") pod \"maas-api-revoke-sub-178f1555-t8brh\" (UID: \"c13290d1-0bf7-40dc-9e78-6e450afbda03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"
Oct 29 22:03:26.935759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.935724    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwgq\" (UniqueName: \"kubernetes.io/projected/c13290d1-0bf7-40dc-9e78-6e450afbda03-kube-api-access-2dwgq\") pod \"maas-api-revoke-sub-178f1555-t8brh\" (UID: \"c13290d1-0bf7-40dc-9e78-6e450afbda03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"
Oct 29 22:03:26.935931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.935862    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-178f1555-t8brh\" (UID: \"c13290d1-0bf7-40dc-9e78-6e450afbda03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"
Oct 29 22:03:26.936006 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.935990    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-178f1555-t8brh\" (UID: \"c13290d1-0bf7-40dc-9e78-6e450afbda03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"
Oct 29 22:03:26.936672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.936648    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-178f1555-t8brh\" (UID: \"c13290d1-0bf7-40dc-9e78-6e450afbda03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"
Oct 29 22:03:26.938388 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.938367    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-178f1555-t8brh\" (UID: \"c13290d1-0bf7-40dc-9e78-6e450afbda03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"
Oct 29 22:03:26.959552 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:26.959528    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwgq\" (UniqueName: \"kubernetes.io/projected/c13290d1-0bf7-40dc-9e78-6e450afbda03-kube-api-access-2dwgq\") pod \"maas-api-revoke-sub-178f1555-t8brh\" (UID: \"c13290d1-0bf7-40dc-9e78-6e450afbda03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"
Oct 29 22:03:27.057524 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:27.057497    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"
Oct 29 22:03:27.380242 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:03:27.380214    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13290d1_0bf7_40dc_9e78_6e450afbda03.slice/crio-5261a4bae016d11cf33a6a0a6bd1175413cb45760132fec1af4e20cfdf384fdb WatchSource:0}: Error finding container 5261a4bae016d11cf33a6a0a6bd1175413cb45760132fec1af4e20cfdf384fdb: Status 404 returned error can't find the container with id 5261a4bae016d11cf33a6a0a6bd1175413cb45760132fec1af4e20cfdf384fdb
Oct 29 22:03:27.381383 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:27.381346    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"]
Oct 29 22:03:27.475423 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:27.475401    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:03:27.475896 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:27.475872    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:03:27.476513 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:27.476496    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:03:28.255179 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:28.255144    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh" event={"ID":"c13290d1-0bf7-40dc-9e78-6e450afbda03","Type":"ContainerStarted","Data":"fe9b4c9b13beaac0f41c209b78d27f8ca5b40b1578333f18131e6fc5e6e96eab"}
Oct 29 22:03:28.255653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:28.255200    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh" event={"ID":"c13290d1-0bf7-40dc-9e78-6e450afbda03","Type":"ContainerStarted","Data":"5261a4bae016d11cf33a6a0a6bd1175413cb45760132fec1af4e20cfdf384fdb"}
Oct 29 22:03:28.301159 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:28.301113    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh" podStartSLOduration=2.3010989139999998 podStartE2EDuration="2.301098914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:28.297971958 +0000 UTC m=+980.798014258" watchObservedRunningTime="2026-10-29 22:03:28.301098914 +0000 UTC m=+980.801141213"
Oct 29 22:03:28.416090 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:03:28.416054    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:03:29.347060 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:29.341432    7082 generic.go:356] "Generic (PLEG): container finished" podID="c13290d1-0bf7-40dc-9e78-6e450afbda03" containerID="fe9b4c9b13beaac0f41c209b78d27f8ca5b40b1578333f18131e6fc5e6e96eab" exitCode=0
Oct 29 22:03:29.347060 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:29.343048    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh" event={"ID":"c13290d1-0bf7-40dc-9e78-6e450afbda03","Type":"ContainerDied","Data":"fe9b4c9b13beaac0f41c209b78d27f8ca5b40b1578333f18131e6fc5e6e96eab"}
Oct 29 22:03:30.526224 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.526192    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"
Oct 29 22:03:30.646091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.646003    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-openshift-ca\") pod \"c13290d1-0bf7-40dc-9e78-6e450afbda03\" (UID: \"c13290d1-0bf7-40dc-9e78-6e450afbda03\") "
Oct 29 22:03:30.646091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.646071    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-xks-ca\") pod \"c13290d1-0bf7-40dc-9e78-6e450afbda03\" (UID: \"c13290d1-0bf7-40dc-9e78-6e450afbda03\") "
Oct 29 22:03:30.646329 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.646227    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13290d1-0bf7-40dc-9e78-6e450afbda03-kube-api-access-2dwgq\" (UniqueName: \"kubernetes.io/projected/c13290d1-0bf7-40dc-9e78-6e450afbda03-kube-api-access-2dwgq\") pod \"c13290d1-0bf7-40dc-9e78-6e450afbda03\" (UID: \"c13290d1-0bf7-40dc-9e78-6e450afbda03\") "
Oct 29 22:03:30.646430 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.646400    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-openshift-ca" pod "c13290d1-0bf7-40dc-9e78-6e450afbda03" (UID: "c13290d1-0bf7-40dc-9e78-6e450afbda03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:03:30.647091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.647071    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:30.648590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.648568    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-xks-ca" pod "c13290d1-0bf7-40dc-9e78-6e450afbda03" (UID: "c13290d1-0bf7-40dc-9e78-6e450afbda03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:03:30.648718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.648586    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13290d1-0bf7-40dc-9e78-6e450afbda03-kube-api-access-2dwgq" pod "c13290d1-0bf7-40dc-9e78-6e450afbda03" (UID: "c13290d1-0bf7-40dc-9e78-6e450afbda03"). InnerVolumeSpecName "kube-api-access-2dwgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:03:30.748716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.748688    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13290d1-0bf7-40dc-9e78-6e450afbda03-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:30.748716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.748713    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dwgq\" (UniqueName: \"kubernetes.io/projected/c13290d1-0bf7-40dc-9e78-6e450afbda03-kube-api-access-2dwgq\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:30.856684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.853533    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"]
Oct 29 22:03:30.856684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.855048    7082 cpu_manager.go:414] "containerMap: removing container" podUID="c13290d1-0bf7-40dc-9e78-6e450afbda03" containerName="revoke-keys"
Oct 29 22:03:30.856684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.855074    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13290d1-0bf7-40dc-9e78-6e450afbda03" containerName="revoke-keys"
Oct 29 22:03:30.856684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.855354    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13290d1-0bf7-40dc-9e78-6e450afbda03" containerName="revoke-keys"
Oct 29 22:03:30.870462 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.868804    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"
Oct 29 22:03:30.900051 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.898875    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"]
Oct 29 22:03:30.980898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.974030    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a3d713-zdmws\" (UID: \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"
Oct 29 22:03:30.980898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.974107    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds8wd\" (UniqueName: \"kubernetes.io/projected/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-kube-api-access-ds8wd\") pod \"maas-api-revoke-sub-37a3d713-zdmws\" (UID: \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"
Oct 29 22:03:30.980898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:30.974145    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a3d713-zdmws\" (UID: \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"
Oct 29 22:03:31.075595 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:31.075561    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds8wd\" (UniqueName: \"kubernetes.io/projected/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-kube-api-access-ds8wd\") pod \"maas-api-revoke-sub-37a3d713-zdmws\" (UID: \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"
Oct 29 22:03:31.075821 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:31.075605    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a3d713-zdmws\" (UID: \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"
Oct 29 22:03:31.075945 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:31.075929    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a3d713-zdmws\" (UID: \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"
Oct 29 22:03:31.076505 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:31.076485    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37a3d713-zdmws\" (UID: \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"
Oct 29 22:03:31.078212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:31.078191    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37a3d713-zdmws\" (UID: \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"
Oct 29 22:03:31.103363 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:31.103325    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds8wd\" (UniqueName: \"kubernetes.io/projected/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-kube-api-access-ds8wd\") pod \"maas-api-revoke-sub-37a3d713-zdmws\" (UID: \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"
Oct 29 22:03:31.272260 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:31.272186    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"
Oct 29 22:03:31.359894 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:31.359862    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh" event={"ID":"c13290d1-0bf7-40dc-9e78-6e450afbda03","Type":"ContainerDied","Data":"5261a4bae016d11cf33a6a0a6bd1175413cb45760132fec1af4e20cfdf384fdb"}
Oct 29 22:03:31.360060 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:31.359922    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5261a4bae016d11cf33a6a0a6bd1175413cb45760132fec1af4e20cfdf384fdb"
Oct 29 22:03:31.360060 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:31.359993    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"
Oct 29 22:03:31.588749 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:31.588719    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"]
Oct 29 22:03:31.589971 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:03:31.589942    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1649f1_1109_40e1_bd32_cc2d6492d4e9.slice/crio-9de41fafc7c814b3a7518ae6dd0695b8bf934bf89d3a4a4a2fb91e4104749f47 WatchSource:0}: Error finding container 9de41fafc7c814b3a7518ae6dd0695b8bf934bf89d3a4a4a2fb91e4104749f47: Status 404 returned error can't find the container with id 9de41fafc7c814b3a7518ae6dd0695b8bf934bf89d3a4a4a2fb91e4104749f47
Oct 29 22:03:32.371572 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:32.371535    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws" event={"ID":"3f1649f1-1109-40e1-bd32-cc2d6492d4e9","Type":"ContainerStarted","Data":"0cf9ffd080269e9a7546f34b822a3e849212925fa3a22032693be47af9d1bfc9"}
Oct 29 22:03:32.371758 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:32.371598    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws" event={"ID":"3f1649f1-1109-40e1-bd32-cc2d6492d4e9","Type":"ContainerStarted","Data":"9de41fafc7c814b3a7518ae6dd0695b8bf934bf89d3a4a4a2fb91e4104749f47"}
Oct 29 22:03:32.417296 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:32.417246    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws" podStartSLOduration=2.41723344 podStartE2EDuration="2.41723344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:32.412805318 +0000 UTC m=+984.912847620" watchObservedRunningTime="2026-10-29 22:03:32.41723344 +0000 UTC m=+984.917275758"
Oct 29 22:03:33.381298 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:33.381263    7082 generic.go:356] "Generic (PLEG): container finished" podID="3f1649f1-1109-40e1-bd32-cc2d6492d4e9" containerID="0cf9ffd080269e9a7546f34b822a3e849212925fa3a22032693be47af9d1bfc9" exitCode=0
Oct 29 22:03:33.381682 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:33.381303    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws" event={"ID":"3f1649f1-1109-40e1-bd32-cc2d6492d4e9","Type":"ContainerDied","Data":"0cf9ffd080269e9a7546f34b822a3e849212925fa3a22032693be47af9d1bfc9"}
Oct 29 22:03:34.201206 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:34.201139    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"
Oct 29 22:03:34.595026 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:34.594998    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"
Oct 29 22:03:34.746109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:34.746020    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-openshift-ca\") pod \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\" (UID: \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\") "
Oct 29 22:03:34.746269 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:34.746166    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-kube-api-access-ds8wd\" (UniqueName: \"kubernetes.io/projected/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-kube-api-access-ds8wd\") pod \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\" (UID: \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\") "
Oct 29 22:03:34.746407 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:34.746385    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-openshift-ca" pod "3f1649f1-1109-40e1-bd32-cc2d6492d4e9" (UID: "3f1649f1-1109-40e1-bd32-cc2d6492d4e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:03:34.746465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:34.746388    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-xks-ca\") pod \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\" (UID: \"3f1649f1-1109-40e1-bd32-cc2d6492d4e9\") "
Oct 29 22:03:34.747727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:34.747685    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:34.748725 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:34.748704    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-xks-ca" pod "3f1649f1-1109-40e1-bd32-cc2d6492d4e9" (UID: "3f1649f1-1109-40e1-bd32-cc2d6492d4e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:03:34.748951 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:34.748937    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-kube-api-access-ds8wd" pod "3f1649f1-1109-40e1-bd32-cc2d6492d4e9" (UID: "3f1649f1-1109-40e1-bd32-cc2d6492d4e9"). InnerVolumeSpecName "kube-api-access-ds8wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:03:34.848857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:34.848823    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:34.848857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:34.848850    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds8wd\" (UniqueName: \"kubernetes.io/projected/3f1649f1-1109-40e1-bd32-cc2d6492d4e9-kube-api-access-ds8wd\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:35.402305 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:35.402277    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"
Oct 29 22:03:35.402518 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:35.402350    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws" event={"ID":"3f1649f1-1109-40e1-bd32-cc2d6492d4e9","Type":"ContainerDied","Data":"9de41fafc7c814b3a7518ae6dd0695b8bf934bf89d3a4a4a2fb91e4104749f47"}
Oct 29 22:03:35.402518 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:35.402396    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9de41fafc7c814b3a7518ae6dd0695b8bf934bf89d3a4a4a2fb91e4104749f47"
Oct 29 22:03:41.188020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.187977    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"]
Oct 29 22:03:41.189152 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.189128    7082 cpu_manager.go:414] "containerMap: removing container" podUID="3f1649f1-1109-40e1-bd32-cc2d6492d4e9" containerName="revoke-keys"
Oct 29 22:03:41.189152 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.189151    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1649f1-1109-40e1-bd32-cc2d6492d4e9" containerName="revoke-keys"
Oct 29 22:03:41.189385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.189373    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1649f1-1109-40e1-bd32-cc2d6492d4e9" containerName="revoke-keys"
Oct 29 22:03:41.192941 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.192923    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"
Oct 29 22:03:41.213750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.213726    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"]
Oct 29 22:03:41.217930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.217903    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:03:41.218031 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.217926    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:03:41.218031 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.217962    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:03:41.219233 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.219202    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:03:41.364031 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.363998    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c992d3a-tqpnc\" (UID: \"4f2f1615-58ef-4aed-9cee-55ee002b4971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"
Oct 29 22:03:41.364208 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.364063    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxqhz\" (UniqueName: \"kubernetes.io/projected/4f2f1615-58ef-4aed-9cee-55ee002b4971-kube-api-access-xxqhz\") pod \"maas-api-revoke-sub-7c992d3a-tqpnc\" (UID: \"4f2f1615-58ef-4aed-9cee-55ee002b4971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"
Oct 29 22:03:41.364260 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.364216    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c992d3a-tqpnc\" (UID: \"4f2f1615-58ef-4aed-9cee-55ee002b4971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"
Oct 29 22:03:41.465784 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.465703    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c992d3a-tqpnc\" (UID: \"4f2f1615-58ef-4aed-9cee-55ee002b4971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"
Oct 29 22:03:41.465784 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.465761    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqhz\" (UniqueName: \"kubernetes.io/projected/4f2f1615-58ef-4aed-9cee-55ee002b4971-kube-api-access-xxqhz\") pod \"maas-api-revoke-sub-7c992d3a-tqpnc\" (UID: \"4f2f1615-58ef-4aed-9cee-55ee002b4971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"
Oct 29 22:03:41.465986 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.465850    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c992d3a-tqpnc\" (UID: \"4f2f1615-58ef-4aed-9cee-55ee002b4971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"
Oct 29 22:03:41.466404 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.466386    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c992d3a-tqpnc\" (UID: \"4f2f1615-58ef-4aed-9cee-55ee002b4971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"
Oct 29 22:03:41.468356 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.468326    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c992d3a-tqpnc\" (UID: \"4f2f1615-58ef-4aed-9cee-55ee002b4971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"
Oct 29 22:03:41.489454 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.489431    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqhz\" (UniqueName: \"kubernetes.io/projected/4f2f1615-58ef-4aed-9cee-55ee002b4971-kube-api-access-xxqhz\") pod \"maas-api-revoke-sub-7c992d3a-tqpnc\" (UID: \"4f2f1615-58ef-4aed-9cee-55ee002b4971\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"
Oct 29 22:03:41.513510 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.513470    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"
Oct 29 22:03:41.848962 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:03:41.848916    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2f1615_58ef_4aed_9cee_55ee002b4971.slice/crio-cb9c3189ec5cee2629e74876dca1cedb634d532f9c9d6324d8ece044c697606f WatchSource:0}: Error finding container cb9c3189ec5cee2629e74876dca1cedb634d532f9c9d6324d8ece044c697606f: Status 404 returned error can't find the container with id cb9c3189ec5cee2629e74876dca1cedb634d532f9c9d6324d8ece044c697606f
Oct 29 22:03:41.849329 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:41.849308    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"]
Oct 29 22:03:42.544593 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:42.543581    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc" event={"ID":"4f2f1615-58ef-4aed-9cee-55ee002b4971","Type":"ContainerStarted","Data":"bbcd32d0c757872de42d79d2f1901f5adc21a804e980e24155106073a784b501"}
Oct 29 22:03:42.544593 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:42.543744    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc" event={"ID":"4f2f1615-58ef-4aed-9cee-55ee002b4971","Type":"ContainerStarted","Data":"cb9c3189ec5cee2629e74876dca1cedb634d532f9c9d6324d8ece044c697606f"}
Oct 29 22:03:42.610716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:42.608066    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc" podStartSLOduration=1.607926634 podStartE2EDuration="1.607926634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:42.602137126 +0000 UTC m=+995.102179427" watchObservedRunningTime="2026-10-29 22:03:42.607926634 +0000 UTC m=+995.107968933"
Oct 29 22:03:43.553260 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:43.553224    7082 generic.go:356] "Generic (PLEG): container finished" podID="4f2f1615-58ef-4aed-9cee-55ee002b4971" containerID="bbcd32d0c757872de42d79d2f1901f5adc21a804e980e24155106073a784b501" exitCode=0
Oct 29 22:03:43.553704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:43.553321    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc" event={"ID":"4f2f1615-58ef-4aed-9cee-55ee002b4971","Type":"ContainerDied","Data":"bbcd32d0c757872de42d79d2f1901f5adc21a804e980e24155106073a784b501"}
Oct 29 22:03:44.760242 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:44.760214    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"
Oct 29 22:03:44.802065 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:44.801963    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-openshift-ca\") pod \"4f2f1615-58ef-4aed-9cee-55ee002b4971\" (UID: \"4f2f1615-58ef-4aed-9cee-55ee002b4971\") "
Oct 29 22:03:44.802616 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:44.802579    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-xks-ca\") pod \"4f2f1615-58ef-4aed-9cee-55ee002b4971\" (UID: \"4f2f1615-58ef-4aed-9cee-55ee002b4971\") "
Oct 29 22:03:44.803036 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:44.802974    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2f1615-58ef-4aed-9cee-55ee002b4971-kube-api-access-xxqhz\" (UniqueName: \"kubernetes.io/projected/4f2f1615-58ef-4aed-9cee-55ee002b4971-kube-api-access-xxqhz\") pod \"4f2f1615-58ef-4aed-9cee-55ee002b4971\" (UID: \"4f2f1615-58ef-4aed-9cee-55ee002b4971\") "
Oct 29 22:03:44.805105 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:44.805082    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-openshift-ca" pod "4f2f1615-58ef-4aed-9cee-55ee002b4971" (UID: "4f2f1615-58ef-4aed-9cee-55ee002b4971"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:03:44.824174 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:44.824099    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2f1615-58ef-4aed-9cee-55ee002b4971-kube-api-access-xxqhz" pod "4f2f1615-58ef-4aed-9cee-55ee002b4971" (UID: "4f2f1615-58ef-4aed-9cee-55ee002b4971"). InnerVolumeSpecName "kube-api-access-xxqhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:03:44.825102 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:44.825067    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-xks-ca" pod "4f2f1615-58ef-4aed-9cee-55ee002b4971" (UID: "4f2f1615-58ef-4aed-9cee-55ee002b4971"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:03:44.905424 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:44.905387    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:44.905424 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:44.905420    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2f1615-58ef-4aed-9cee-55ee002b4971-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:44.905424 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:44.905429    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxqhz\" (UniqueName: \"kubernetes.io/projected/4f2f1615-58ef-4aed-9cee-55ee002b4971-kube-api-access-xxqhz\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:45.572938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:45.572904    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"
Oct 29 22:03:45.572938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:45.572934    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc" event={"ID":"4f2f1615-58ef-4aed-9cee-55ee002b4971","Type":"ContainerDied","Data":"cb9c3189ec5cee2629e74876dca1cedb634d532f9c9d6324d8ece044c697606f"}
Oct 29 22:03:45.573172 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:45.572980    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb9c3189ec5cee2629e74876dca1cedb634d532f9c9d6324d8ece044c697606f"
Oct 29 22:03:50.038647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:50.038590    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"]
Oct 29 22:03:50.050135 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:50.050096    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44867098-f8zhg"]
Oct 29 22:03:51.041847 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:51.041812    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"]
Oct 29 22:03:51.052766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:51.052742    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b337d2-wg7lb"]
Oct 29 22:03:51.639036 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:51.638998    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a110df2-476c-4d68-ab62-5605e00cdeee" path="/var/lib/kubelet/pods/2a110df2-476c-4d68-ab62-5605e00cdeee/volumes"
Oct 29 22:03:51.639985 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:51.639956    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="360c6c19-fe30-47d9-8bc6-3e7fe73859c1" path="/var/lib/kubelet/pods/360c6c19-fe30-47d9-8bc6-3e7fe73859c1/volumes"
Oct 29 22:03:56.059147 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.059111    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"]
Oct 29 22:03:56.065086 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.065052    7082 cpu_manager.go:414] "containerMap: removing container" podUID="4f2f1615-58ef-4aed-9cee-55ee002b4971" containerName="revoke-keys"
Oct 29 22:03:56.065302 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.065287    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2f1615-58ef-4aed-9cee-55ee002b4971" containerName="revoke-keys"
Oct 29 22:03:56.070233 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.070207    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2f1615-58ef-4aed-9cee-55ee002b4971" containerName="revoke-keys"
Oct 29 22:03:56.079092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.079069    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"
Oct 29 22:03:56.089275 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.089243    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"]
Oct 29 22:03:56.110033 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.109997    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:03:56.110033 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.109997    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:03:56.110253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.110094    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:03:56.110253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.110169    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:03:56.175958 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.175917    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a15f6f1-f96b2\" (UID: \"7e31289b-2e61-4248-85d4-7578914b2cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"
Oct 29 22:03:56.176131 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.175996    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894fb\" (UniqueName: \"kubernetes.io/projected/7e31289b-2e61-4248-85d4-7578914b2cf7-kube-api-access-894fb\") pod \"maas-api-revoke-sub-9a15f6f1-f96b2\" (UID: \"7e31289b-2e61-4248-85d4-7578914b2cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"
Oct 29 22:03:56.176304 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.176287    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a15f6f1-f96b2\" (UID: \"7e31289b-2e61-4248-85d4-7578914b2cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"
Oct 29 22:03:56.278156 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.278127    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a15f6f1-f96b2\" (UID: \"7e31289b-2e61-4248-85d4-7578914b2cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"
Oct 29 22:03:56.278316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.278177    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a15f6f1-f96b2\" (UID: \"7e31289b-2e61-4248-85d4-7578914b2cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"
Oct 29 22:03:56.278316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.278227    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894fb\" (UniqueName: \"kubernetes.io/projected/7e31289b-2e61-4248-85d4-7578914b2cf7-kube-api-access-894fb\") pod \"maas-api-revoke-sub-9a15f6f1-f96b2\" (UID: \"7e31289b-2e61-4248-85d4-7578914b2cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"
Oct 29 22:03:56.278862 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.278840    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a15f6f1-f96b2\" (UID: \"7e31289b-2e61-4248-85d4-7578914b2cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"
Oct 29 22:03:56.280687 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.280617    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a15f6f1-f96b2\" (UID: \"7e31289b-2e61-4248-85d4-7578914b2cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"
Oct 29 22:03:56.301461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.301430    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-894fb\" (UniqueName: \"kubernetes.io/projected/7e31289b-2e61-4248-85d4-7578914b2cf7-kube-api-access-894fb\") pod \"maas-api-revoke-sub-9a15f6f1-f96b2\" (UID: \"7e31289b-2e61-4248-85d4-7578914b2cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"
Oct 29 22:03:56.408601 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.408528    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"
Oct 29 22:03:56.723403 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:03:56.723371    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e31289b_2e61_4248_85d4_7578914b2cf7.slice/crio-1fb924b6f31205aa5c64ada5f0fa7a4b93852885cec23781a7fd99fef8a30c3e WatchSource:0}: Error finding container 1fb924b6f31205aa5c64ada5f0fa7a4b93852885cec23781a7fd99fef8a30c3e: Status 404 returned error can't find the container with id 1fb924b6f31205aa5c64ada5f0fa7a4b93852885cec23781a7fd99fef8a30c3e
Oct 29 22:03:56.723794 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.723772    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"]
Oct 29 22:03:56.777372 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:56.777343    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2" event={"ID":"7e31289b-2e61-4248-85d4-7578914b2cf7","Type":"ContainerStarted","Data":"1fb924b6f31205aa5c64ada5f0fa7a4b93852885cec23781a7fd99fef8a30c3e"}
Oct 29 22:03:57.874059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:57.874020    7082 generic.go:356] "Generic (PLEG): container finished" podID="7e31289b-2e61-4248-85d4-7578914b2cf7" containerID="b57e57d9dc87faf6187a1df1439f82967ecd7beb338a81db6ccba1f4a0db12ed" exitCode=0
Oct 29 22:03:57.874799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:57.874197    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2" event={"ID":"7e31289b-2e61-4248-85d4-7578914b2cf7","Type":"ContainerDied","Data":"b57e57d9dc87faf6187a1df1439f82967ecd7beb338a81db6ccba1f4a0db12ed"}
Oct 29 22:03:58.413526 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:03:58.413484    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:03:59.102933 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:59.102909    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"
Oct 29 22:03:59.240036 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:59.239937    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e31289b-2e61-4248-85d4-7578914b2cf7-kube-api-access-894fb\" (UniqueName: \"kubernetes.io/projected/7e31289b-2e61-4248-85d4-7578914b2cf7-kube-api-access-894fb\") pod \"7e31289b-2e61-4248-85d4-7578914b2cf7\" (UID: \"7e31289b-2e61-4248-85d4-7578914b2cf7\") "
Oct 29 22:03:59.240036 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:59.240010    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-openshift-ca\") pod \"7e31289b-2e61-4248-85d4-7578914b2cf7\" (UID: \"7e31289b-2e61-4248-85d4-7578914b2cf7\") "
Oct 29 22:03:59.240276 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:59.240202    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-xks-ca\") pod \"7e31289b-2e61-4248-85d4-7578914b2cf7\" (UID: \"7e31289b-2e61-4248-85d4-7578914b2cf7\") "
Oct 29 22:03:59.240351 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:59.240324    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-openshift-ca" pod "7e31289b-2e61-4248-85d4-7578914b2cf7" (UID: "7e31289b-2e61-4248-85d4-7578914b2cf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:03:59.240863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:59.240843    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:59.242327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:59.242297    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e31289b-2e61-4248-85d4-7578914b2cf7-kube-api-access-894fb" pod "7e31289b-2e61-4248-85d4-7578914b2cf7" (UID: "7e31289b-2e61-4248-85d4-7578914b2cf7"). InnerVolumeSpecName "kube-api-access-894fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:03:59.242327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:59.242320    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-xks-ca" pod "7e31289b-2e61-4248-85d4-7578914b2cf7" (UID: "7e31289b-2e61-4248-85d4-7578914b2cf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:03:59.342880 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:59.342853    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-894fb\" (UniqueName: \"kubernetes.io/projected/7e31289b-2e61-4248-85d4-7578914b2cf7-kube-api-access-894fb\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:59.342880 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:59.342877    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e31289b-2e61-4248-85d4-7578914b2cf7-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:03:59.896666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:59.896559    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"
Oct 29 22:03:59.896853 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:59.896558    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2" event={"ID":"7e31289b-2e61-4248-85d4-7578914b2cf7","Type":"ContainerDied","Data":"1fb924b6f31205aa5c64ada5f0fa7a4b93852885cec23781a7fd99fef8a30c3e"}
Oct 29 22:03:59.896853 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:03:59.896735    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fb924b6f31205aa5c64ada5f0fa7a4b93852885cec23781a7fd99fef8a30c3e"
Oct 29 22:04:01.329190 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.329158    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"]
Oct 29 22:04:01.329961 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.329945    7082 cpu_manager.go:414] "containerMap: removing container" podUID="7e31289b-2e61-4248-85d4-7578914b2cf7" containerName="revoke-keys"
Oct 29 22:04:01.330035 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.329962    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e31289b-2e61-4248-85d4-7578914b2cf7" containerName="revoke-keys"
Oct 29 22:04:01.330081 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.330075    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e31289b-2e61-4248-85d4-7578914b2cf7" containerName="revoke-keys"
Oct 29 22:04:01.334161 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.334141    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"
Oct 29 22:04:01.356981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.356953    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"]
Oct 29 22:04:01.361499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.361473    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:04:01.361653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.361534    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:04:01.383915 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.383880    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af235450-f751-4529-9eb7-4493375ae287-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5\" (UID: \"af235450-f751-4529-9eb7-4493375ae287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"
Oct 29 22:04:01.384062 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.384010    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng7zf\" (UniqueName: \"kubernetes.io/projected/af235450-f751-4529-9eb7-4493375ae287-kube-api-access-ng7zf\") pod \"maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5\" (UID: \"af235450-f751-4529-9eb7-4493375ae287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"
Oct 29 22:04:01.384276 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.384259    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af235450-f751-4529-9eb7-4493375ae287-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5\" (UID: \"af235450-f751-4529-9eb7-4493375ae287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"
Oct 29 22:04:01.394567 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.394543    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:04:01.394699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.394620    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:04:01.485986 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.485946    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af235450-f751-4529-9eb7-4493375ae287-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5\" (UID: \"af235450-f751-4529-9eb7-4493375ae287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"
Oct 29 22:04:01.486168 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.486087    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af235450-f751-4529-9eb7-4493375ae287-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5\" (UID: \"af235450-f751-4529-9eb7-4493375ae287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"
Oct 29 22:04:01.486168 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.486139    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng7zf\" (UniqueName: \"kubernetes.io/projected/af235450-f751-4529-9eb7-4493375ae287-kube-api-access-ng7zf\") pod \"maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5\" (UID: \"af235450-f751-4529-9eb7-4493375ae287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"
Oct 29 22:04:01.486808 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.486782    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af235450-f751-4529-9eb7-4493375ae287-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5\" (UID: \"af235450-f751-4529-9eb7-4493375ae287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"
Oct 29 22:04:01.488740 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.488707    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af235450-f751-4529-9eb7-4493375ae287-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5\" (UID: \"af235450-f751-4529-9eb7-4493375ae287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"
Oct 29 22:04:01.512211 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.512185    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng7zf\" (UniqueName: \"kubernetes.io/projected/af235450-f751-4529-9eb7-4493375ae287-kube-api-access-ng7zf\") pod \"maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5\" (UID: \"af235450-f751-4529-9eb7-4493375ae287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"
Oct 29 22:04:01.649400 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.649322    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"
Oct 29 22:04:01.910174 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.910086    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"]
Oct 29 22:04:01.926374 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.926340    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:01.946410 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.946381    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"]
Oct 29 22:04:01.950705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.950675    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-734480-openshift-default-dockercfg-6xpmh\""
Oct 29 22:04:01.985094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.985066    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"]
Oct 29 22:04:01.993923 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:04:01.993894    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf235450_f751_4529_9eb7_4493375ae287.slice/crio-217dd91c0d7232f8579f6c0b3a196367e228a995a32fe527b95db7d135e92d3b WatchSource:0}: Error finding container 217dd91c0d7232f8579f6c0b3a196367e228a995a32fe527b95db7d135e92d3b: Status 404 returned error can't find the container with id 217dd91c0d7232f8579f6c0b3a196367e228a995a32fe527b95db7d135e92d3b
Oct 29 22:04:01.994489 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.994459    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-podinfo\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:01.994830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.994715    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-token\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:01.994942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.994923    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-certs\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:01.995006 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.994972    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-credential-socket\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:01.995109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.995082    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-envoy\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:01.995153 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.995141    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-data\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:01.995236 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.995220    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4656f\" (UniqueName: \"kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-kube-api-access-4656f\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:01.995281 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.995266    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-socket\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:01.995511 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:01.995493    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423fc4e9-6575-4690-9263-83ffff40dfc3-istiod-ca-cert\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.113759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.112883    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423fc4e9-6575-4690-9263-83ffff40dfc3-istiod-ca-cert\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.444496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.444433    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-podinfo\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.445939 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.445907    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-token\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.446481 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.446461    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-certs\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.447133 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.447111    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-credential-socket\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.447598 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.447582    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-envoy\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.478656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.470793    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-data\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.478656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.470920    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4656f\" (UniqueName: \"kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-kube-api-access-4656f\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.478656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.470971    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-socket\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.478656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.471643    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-socket\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.478656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.126052    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423fc4e9-6575-4690-9263-83ffff40dfc3-istiod-ca-cert\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.478656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.471979    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-data\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.478656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.447508    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-credential-socket\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.478656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.447034    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-certs\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.501259 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.501227    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-envoy\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.502515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.502485    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-podinfo\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.511612 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.511217    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-token\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.546588 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.546554    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4656f\" (UniqueName: \"kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-kube-api-access-4656f\") pod \"e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") " pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.563233 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.557039    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:02.919592 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.919563    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"]
Oct 29 22:04:02.919717 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:04:02.919575    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423fc4e9_6575_4690_9263_83ffff40dfc3.slice/crio-724b7b542d9dbbc1d36181c9b33f8d96d8cda68f19bb07a2a35a9da3ce6c1377 WatchSource:0}: Error finding container 724b7b542d9dbbc1d36181c9b33f8d96d8cda68f19bb07a2a35a9da3ce6c1377: Status 404 returned error can't find the container with id 724b7b542d9dbbc1d36181c9b33f8d96d8cda68f19bb07a2a35a9da3ce6c1377
Oct 29 22:04:02.922146 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.922113    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:04:02.922255 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.922195    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:04:02.922255 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.922242    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:04:02.950150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.950115    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8" event={"ID":"423fc4e9-6575-4690-9263-83ffff40dfc3","Type":"ContainerStarted","Data":"724b7b542d9dbbc1d36181c9b33f8d96d8cda68f19bb07a2a35a9da3ce6c1377"}
Oct 29 22:04:02.951906 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:02.951880    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5" event={"ID":"af235450-f751-4529-9eb7-4493375ae287","Type":"ContainerStarted","Data":"217dd91c0d7232f8579f6c0b3a196367e228a995a32fe527b95db7d135e92d3b"}
Oct 29 22:04:03.205705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.202679    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"]
Oct 29 22:04:03.255030 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.254995    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"]
Oct 29 22:04:03.255264 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.255170    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"
Oct 29 22:04:03.322614 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.322177    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96964222-318a-433e-b83c-dfff82598b9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc875add-czt7c\" (UID: \"96964222-318a-433e-b83c-dfff82598b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"
Oct 29 22:04:03.322614 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.322319    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96964222-318a-433e-b83c-dfff82598b9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc875add-czt7c\" (UID: \"96964222-318a-433e-b83c-dfff82598b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"
Oct 29 22:04:03.322614 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.322451    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scs6n\" (UniqueName: \"kubernetes.io/projected/96964222-318a-433e-b83c-dfff82598b9a-kube-api-access-scs6n\") pod \"maas-api-revoke-sub-cc875add-czt7c\" (UID: \"96964222-318a-433e-b83c-dfff82598b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"
Oct 29 22:04:03.423956 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.423926    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96964222-318a-433e-b83c-dfff82598b9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc875add-czt7c\" (UID: \"96964222-318a-433e-b83c-dfff82598b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"
Oct 29 22:04:03.424140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.424095    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96964222-318a-433e-b83c-dfff82598b9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc875add-czt7c\" (UID: \"96964222-318a-433e-b83c-dfff82598b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"
Oct 29 22:04:03.424268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.424252    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scs6n\" (UniqueName: \"kubernetes.io/projected/96964222-318a-433e-b83c-dfff82598b9a-kube-api-access-scs6n\") pod \"maas-api-revoke-sub-cc875add-czt7c\" (UID: \"96964222-318a-433e-b83c-dfff82598b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"
Oct 29 22:04:03.424832 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.424802    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96964222-318a-433e-b83c-dfff82598b9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc875add-czt7c\" (UID: \"96964222-318a-433e-b83c-dfff82598b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"
Oct 29 22:04:03.426644 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.426604    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96964222-318a-433e-b83c-dfff82598b9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc875add-czt7c\" (UID: \"96964222-318a-433e-b83c-dfff82598b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"
Oct 29 22:04:03.447879 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.447853    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scs6n\" (UniqueName: \"kubernetes.io/projected/96964222-318a-433e-b83c-dfff82598b9a-kube-api-access-scs6n\") pod \"maas-api-revoke-sub-cc875add-czt7c\" (UID: \"96964222-318a-433e-b83c-dfff82598b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"
Oct 29 22:04:03.578400 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.578364    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"
Oct 29 22:04:03.898393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.898361    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"]
Oct 29 22:04:03.898774 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:04:03.898750    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96964222_318a_433e_b83c_dfff82598b9a.slice/crio-f028c267474373ceeac01b91e687b1f8e444ee3b193dec9f69132453b1a06d27 WatchSource:0}: Error finding container f028c267474373ceeac01b91e687b1f8e444ee3b193dec9f69132453b1a06d27: Status 404 returned error can't find the container with id f028c267474373ceeac01b91e687b1f8e444ee3b193dec9f69132453b1a06d27
Oct 29 22:04:03.963572 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.963542    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8" event={"ID":"423fc4e9-6575-4690-9263-83ffff40dfc3","Type":"ContainerStarted","Data":"dc7bd3c00431b323bdc0ba5c5d37e12df664a7bccaed4048dfededf10ab964dd"}
Oct 29 22:04:03.965108 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.965085    7082 generic.go:356] "Generic (PLEG): container finished" podID="af235450-f751-4529-9eb7-4493375ae287" containerID="0bb1d54e8f31c6949786dbbbfb1ac012b410a38ee011d22c599105597d4b62a2" exitCode=0
Oct 29 22:04:03.965207 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.965159    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5" event={"ID":"af235450-f751-4529-9eb7-4493375ae287","Type":"ContainerDied","Data":"0bb1d54e8f31c6949786dbbbfb1ac012b410a38ee011d22c599105597d4b62a2"}
Oct 29 22:04:03.966431 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:03.966408    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c" event={"ID":"96964222-318a-433e-b83c-dfff82598b9a","Type":"ContainerStarted","Data":"f028c267474373ceeac01b91e687b1f8e444ee3b193dec9f69132453b1a06d27"}
Oct 29 22:04:04.010888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:04.010843    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8" podStartSLOduration=3.010829463 podStartE2EDuration="3.010829463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:04.009459018 +0000 UTC m=+1016.509501332" watchObservedRunningTime="2026-10-29 22:04:04.010829463 +0000 UTC m=+1016.510871762"
Oct 29 22:04:04.568263 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:04.568175    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:04.573363 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:04.573334    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:04.979343 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:04.979306    7082 generic.go:356] "Generic (PLEG): container finished" podID="96964222-318a-433e-b83c-dfff82598b9a" containerID="b502183142e9f7874b53527e908cfb6d7448b9fcf58eb11b0195e74c3aef5dc1" exitCode=0
Oct 29 22:04:04.979490 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:04.979382    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c" event={"ID":"96964222-318a-433e-b83c-dfff82598b9a","Type":"ContainerDied","Data":"b502183142e9f7874b53527e908cfb6d7448b9fcf58eb11b0195e74c3aef5dc1"}
Oct 29 22:04:04.979836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:04.979820    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:04.980778 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:04.980749    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:04:05.192442 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:05.191755    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"
Oct 29 22:04:05.267266 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:05.267235    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af235450-f751-4529-9eb7-4493375ae287-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af235450-f751-4529-9eb7-4493375ae287-maas-api-xks-ca\") pod \"af235450-f751-4529-9eb7-4493375ae287\" (UID: \"af235450-f751-4529-9eb7-4493375ae287\") "
Oct 29 22:04:05.267442 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:05.267370    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af235450-f751-4529-9eb7-4493375ae287-kube-api-access-ng7zf\" (UniqueName: \"kubernetes.io/projected/af235450-f751-4529-9eb7-4493375ae287-kube-api-access-ng7zf\") pod \"af235450-f751-4529-9eb7-4493375ae287\" (UID: \"af235450-f751-4529-9eb7-4493375ae287\") "
Oct 29 22:04:05.267516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:05.267449    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af235450-f751-4529-9eb7-4493375ae287-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af235450-f751-4529-9eb7-4493375ae287-maas-api-openshift-ca\") pod \"af235450-f751-4529-9eb7-4493375ae287\" (UID: \"af235450-f751-4529-9eb7-4493375ae287\") "
Oct 29 22:04:05.268181 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:05.268149    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af235450-f751-4529-9eb7-4493375ae287-maas-api-openshift-ca" pod "af235450-f751-4529-9eb7-4493375ae287" (UID: "af235450-f751-4529-9eb7-4493375ae287"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:04:05.268322 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:05.268301    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af235450-f751-4529-9eb7-4493375ae287-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:04:05.270197 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:05.270164    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af235450-f751-4529-9eb7-4493375ae287-kube-api-access-ng7zf" pod "af235450-f751-4529-9eb7-4493375ae287" (UID: "af235450-f751-4529-9eb7-4493375ae287"). InnerVolumeSpecName "kube-api-access-ng7zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:04:05.270299 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:05.270202    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af235450-f751-4529-9eb7-4493375ae287-maas-api-xks-ca" pod "af235450-f751-4529-9eb7-4493375ae287" (UID: "af235450-f751-4529-9eb7-4493375ae287"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:04:05.369368 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:05.369343    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng7zf\" (UniqueName: \"kubernetes.io/projected/af235450-f751-4529-9eb7-4493375ae287-kube-api-access-ng7zf\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:04:05.369368 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:05.369367    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af235450-f751-4529-9eb7-4493375ae287-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:04:05.988921 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:05.988832    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"
Oct 29 22:04:05.988921 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:05.988861    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5" event={"ID":"af235450-f751-4529-9eb7-4493375ae287","Type":"ContainerDied","Data":"217dd91c0d7232f8579f6c0b3a196367e228a995a32fe527b95db7d135e92d3b"}
Oct 29 22:04:05.988921 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:05.988914    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="217dd91c0d7232f8579f6c0b3a196367e228a995a32fe527b95db7d135e92d3b"
Oct 29 22:04:06.171710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:06.171688    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"
Oct 29 22:04:06.180845 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:06.180822    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96964222-318a-433e-b83c-dfff82598b9a-kube-api-access-scs6n\" (UniqueName: \"kubernetes.io/projected/96964222-318a-433e-b83c-dfff82598b9a-kube-api-access-scs6n\") pod \"96964222-318a-433e-b83c-dfff82598b9a\" (UID: \"96964222-318a-433e-b83c-dfff82598b9a\") "
Oct 29 22:04:06.180980 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:06.180966    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96964222-318a-433e-b83c-dfff82598b9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96964222-318a-433e-b83c-dfff82598b9a-maas-api-openshift-ca\") pod \"96964222-318a-433e-b83c-dfff82598b9a\" (UID: \"96964222-318a-433e-b83c-dfff82598b9a\") "
Oct 29 22:04:06.181042 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:06.181025    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96964222-318a-433e-b83c-dfff82598b9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96964222-318a-433e-b83c-dfff82598b9a-maas-api-xks-ca\") pod \"96964222-318a-433e-b83c-dfff82598b9a\" (UID: \"96964222-318a-433e-b83c-dfff82598b9a\") "
Oct 29 22:04:06.181451 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:06.181387    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96964222-318a-433e-b83c-dfff82598b9a-maas-api-openshift-ca" pod "96964222-318a-433e-b83c-dfff82598b9a" (UID: "96964222-318a-433e-b83c-dfff82598b9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:04:06.181766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:06.181741    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96964222-318a-433e-b83c-dfff82598b9a-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:04:06.182990 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:06.182969    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96964222-318a-433e-b83c-dfff82598b9a-kube-api-access-scs6n" pod "96964222-318a-433e-b83c-dfff82598b9a" (UID: "96964222-318a-433e-b83c-dfff82598b9a"). InnerVolumeSpecName "kube-api-access-scs6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:04:06.183341 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:06.183315    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96964222-318a-433e-b83c-dfff82598b9a-maas-api-xks-ca" pod "96964222-318a-433e-b83c-dfff82598b9a" (UID: "96964222-318a-433e-b83c-dfff82598b9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:04:06.283245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:06.283168    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96964222-318a-433e-b83c-dfff82598b9a-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:04:06.283245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:06.283200    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scs6n\" (UniqueName: \"kubernetes.io/projected/96964222-318a-433e-b83c-dfff82598b9a-kube-api-access-scs6n\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:04:07.003688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:07.003234    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"
Oct 29 22:04:07.003688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:07.003517    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c" event={"ID":"96964222-318a-433e-b83c-dfff82598b9a","Type":"ContainerDied","Data":"f028c267474373ceeac01b91e687b1f8e444ee3b193dec9f69132453b1a06d27"}
Oct 29 22:04:07.003688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:07.003572    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f028c267474373ceeac01b91e687b1f8e444ee3b193dec9f69132453b1a06d27"
Oct 29 22:04:08.248273 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.248231    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"]
Oct 29 22:04:08.249455 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.249434    7082 cpu_manager.go:414] "containerMap: removing container" podUID="af235450-f751-4529-9eb7-4493375ae287" containerName="revoke-keys"
Oct 29 22:04:08.249512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.249459    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af235450-f751-4529-9eb7-4493375ae287" containerName="revoke-keys"
Oct 29 22:04:08.249512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.249483    7082 cpu_manager.go:414] "containerMap: removing container" podUID="96964222-318a-433e-b83c-dfff82598b9a" containerName="revoke-keys"
Oct 29 22:04:08.249512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.249490    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96964222-318a-433e-b83c-dfff82598b9a" containerName="revoke-keys"
Oct 29 22:04:08.249689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.249676    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="af235450-f751-4529-9eb7-4493375ae287" containerName="revoke-keys"
Oct 29 22:04:08.249730 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.249695    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="96964222-318a-433e-b83c-dfff82598b9a" containerName="revoke-keys"
Oct 29 22:04:08.254521 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.254499    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"
Oct 29 22:04:08.269018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.268991    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-734480\""
Oct 29 22:04:08.271615 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.271591    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"]
Oct 29 22:04:08.298013 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.297983    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"]
Oct 29 22:04:08.303906 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.303851    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"
Oct 29 22:04:08.337002 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.336972    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-734480-dockercfg-k96mf\""
Oct 29 22:04:08.338016 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.337988    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grflk\" (UniqueName: \"kubernetes.io/projected/2a691653-24d1-4772-813b-067bb8f78d97-kube-api-access-grflk\") pod \"payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5\" (UID: \"2a691653-24d1-4772-813b-067bb8f78d97\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"
Oct 29 22:04:08.338150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.338060    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0bfbbf9-27ba-414b-875d-ce9807964aa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc\" (UID: \"a0bfbbf9-27ba-414b-875d-ce9807964aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"
Oct 29 22:04:08.338213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.338173    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m579s\" (UniqueName: \"kubernetes.io/projected/a0bfbbf9-27ba-414b-875d-ce9807964aa4-kube-api-access-m579s\") pod \"payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc\" (UID: \"a0bfbbf9-27ba-414b-875d-ce9807964aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"
Oct 29 22:04:08.338417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.338390    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a691653-24d1-4772-813b-067bb8f78d97-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5\" (UID: \"2a691653-24d1-4772-813b-067bb8f78d97\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"
Oct 29 22:04:08.345292 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.345267    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"]
Oct 29 22:04:08.440337 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.440302    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grflk\" (UniqueName: \"kubernetes.io/projected/2a691653-24d1-4772-813b-067bb8f78d97-kube-api-access-grflk\") pod \"payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5\" (UID: \"2a691653-24d1-4772-813b-067bb8f78d97\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"
Oct 29 22:04:08.440504 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.440355    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0bfbbf9-27ba-414b-875d-ce9807964aa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc\" (UID: \"a0bfbbf9-27ba-414b-875d-ce9807964aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"
Oct 29 22:04:08.440504 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.440389    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m579s\" (UniqueName: \"kubernetes.io/projected/a0bfbbf9-27ba-414b-875d-ce9807964aa4-kube-api-access-m579s\") pod \"payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc\" (UID: \"a0bfbbf9-27ba-414b-875d-ce9807964aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"
Oct 29 22:04:08.440504 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.440490    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a691653-24d1-4772-813b-067bb8f78d97-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5\" (UID: \"2a691653-24d1-4772-813b-067bb8f78d97\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"
Oct 29 22:04:08.441236 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.441216    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a691653-24d1-4772-813b-067bb8f78d97-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5\" (UID: \"2a691653-24d1-4772-813b-067bb8f78d97\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"
Oct 29 22:04:08.441324 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.441214    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0bfbbf9-27ba-414b-875d-ce9807964aa4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc\" (UID: \"a0bfbbf9-27ba-414b-875d-ce9807964aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"
Oct 29 22:04:08.549805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.536450    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m579s\" (UniqueName: \"kubernetes.io/projected/a0bfbbf9-27ba-414b-875d-ce9807964aa4-kube-api-access-m579s\") pod \"payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc\" (UID: \"a0bfbbf9-27ba-414b-875d-ce9807964aa4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"
Oct 29 22:04:08.561810 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.547934    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grflk\" (UniqueName: \"kubernetes.io/projected/2a691653-24d1-4772-813b-067bb8f78d97-kube-api-access-grflk\") pod \"payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5\" (UID: \"2a691653-24d1-4772-813b-067bb8f78d97\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"
Oct 29 22:04:08.592745 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.579279    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"
Oct 29 22:04:08.623319 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.623287    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"
Oct 29 22:04:08.629512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.629485    7082 scope.go:122] "RemoveContainer" containerID="0dea48477c192d880dcbde964bec610d3db6af09ad7ecebd4fdd64ba7bdde27c"
Oct 29 22:04:08.910858 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.909962    7082 scope.go:122] "RemoveContainer" containerID="3a627afe5dc21c6e67d963b7644be36e05fceb458400de57a692a8ba04545a40"
Oct 29 22:04:08.931417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.931390    7082 scope.go:122] "RemoveContainer" containerID="85cbffe183e21c21296dba518043aea9c36fc408cf5f49b20dba4fcbf779c785"
Oct 29 22:04:08.950968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.950812    7082 scope.go:122] "RemoveContainer" containerID="4dad66ae77a60ce438a2bf2495c19cc90e5e15e30cd1e7a3643059076453b0e7"
Oct 29 22:04:08.976897 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:08.974130    7082 scope.go:122] "RemoveContainer" containerID="3cfd18bbbb255d6d36b086fc1475062ce6cb40d9860d7bcae9fd4228a1207a7c"
Oct 29 22:04:09.201348 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:04:09.201308    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a691653_24d1_4772_813b_067bb8f78d97.slice/crio-150aaba882ecd7a3d88bf845ec611f971aa854d4f417c272cd832a47c121ee3f WatchSource:0}: Error finding container 150aaba882ecd7a3d88bf845ec611f971aa854d4f417c272cd832a47c121ee3f: Status 404 returned error can't find the container with id 150aaba882ecd7a3d88bf845ec611f971aa854d4f417c272cd832a47c121ee3f
Oct 29 22:04:09.203689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:09.203655    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"]
Oct 29 22:04:09.223247 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:04:09.223213    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bfbbf9_27ba_414b_875d_ce9807964aa4.slice/crio-ab86762080e27a5bdf54239f8361cdb7b61d335cd67ecc26ac424dcc26a9f5c7 WatchSource:0}: Error finding container ab86762080e27a5bdf54239f8361cdb7b61d335cd67ecc26ac424dcc26a9f5c7: Status 404 returned error can't find the container with id ab86762080e27a5bdf54239f8361cdb7b61d335cd67ecc26ac424dcc26a9f5c7
Oct 29 22:04:09.224601 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:09.224574    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"]
Oct 29 22:04:10.049413 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:10.049374    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc" event={"ID":"a0bfbbf9-27ba-414b-875d-ce9807964aa4","Type":"ContainerStarted","Data":"da47863bf5a3b9fe87b77e6e8d7028ecd068e216dcbcfec4dce69ee7e35231c5"}
Oct 29 22:04:10.049932 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:10.049442    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc" event={"ID":"a0bfbbf9-27ba-414b-875d-ce9807964aa4","Type":"ContainerStarted","Data":"ab86762080e27a5bdf54239f8361cdb7b61d335cd67ecc26ac424dcc26a9f5c7"}
Oct 29 22:04:10.049932 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:10.049473    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"
Oct 29 22:04:10.051430 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:10.051404    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5" event={"ID":"2a691653-24d1-4772-813b-067bb8f78d97","Type":"ContainerStarted","Data":"56ebedb5c9db1e76926b7def250cabe871aa2df0eadf3573603407f50a5f3b48"}
Oct 29 22:04:10.051529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:10.051447    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5" event={"ID":"2a691653-24d1-4772-813b-067bb8f78d97","Type":"ContainerStarted","Data":"150aaba882ecd7a3d88bf845ec611f971aa854d4f417c272cd832a47c121ee3f"}
Oct 29 22:04:10.051529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:10.051466    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"
Oct 29 22:04:10.103567 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:10.103514    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc" podStartSLOduration=2.103498763 podStartE2EDuration="2.103498763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:10.091373589 +0000 UTC m=+1022.591415889" watchObservedRunningTime="2026-10-29 22:04:10.103498763 +0000 UTC m=+1022.603541058"
Oct 29 22:04:10.135620 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:10.135569    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5" podStartSLOduration=2.135551133 podStartE2EDuration="2.135551133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:10.132444633 +0000 UTC m=+1022.632486932" watchObservedRunningTime="2026-10-29 22:04:10.135551133 +0000 UTC m=+1022.635593433"
Oct 29 22:04:13.529213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.529175    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"]
Oct 29 22:04:13.534215 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.534196    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"
Oct 29 22:04:13.548844 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.548821    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:04:13.548988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.548923    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:04:13.555297 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.555269    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"]
Oct 29 22:04:13.562922 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.560826    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:04:13.562922 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.560838    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:04:13.562922 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.561458    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b041902-qfqlp\" (UID: \"1c3563f4-8c85-4267-acf0-020554ae7aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"
Oct 29 22:04:13.562922 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.561570    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nrr5\" (UniqueName: \"kubernetes.io/projected/1c3563f4-8c85-4267-acf0-020554ae7aa2-kube-api-access-7nrr5\") pod \"maas-api-revoke-sub-4b041902-qfqlp\" (UID: \"1c3563f4-8c85-4267-acf0-020554ae7aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"
Oct 29 22:04:13.562922 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.561590    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b041902-qfqlp\" (UID: \"1c3563f4-8c85-4267-acf0-020554ae7aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"
Oct 29 22:04:13.662858 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.662825    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b041902-qfqlp\" (UID: \"1c3563f4-8c85-4267-acf0-020554ae7aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"
Oct 29 22:04:13.663067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.663045    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nrr5\" (UniqueName: \"kubernetes.io/projected/1c3563f4-8c85-4267-acf0-020554ae7aa2-kube-api-access-7nrr5\") pod \"maas-api-revoke-sub-4b041902-qfqlp\" (UID: \"1c3563f4-8c85-4267-acf0-020554ae7aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"
Oct 29 22:04:13.663148 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.663082    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b041902-qfqlp\" (UID: \"1c3563f4-8c85-4267-acf0-020554ae7aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"
Oct 29 22:04:13.663754 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.663731    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b041902-qfqlp\" (UID: \"1c3563f4-8c85-4267-acf0-020554ae7aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"
Oct 29 22:04:13.665346 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.665327    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b041902-qfqlp\" (UID: \"1c3563f4-8c85-4267-acf0-020554ae7aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"
Oct 29 22:04:13.713658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.686792    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nrr5\" (UniqueName: \"kubernetes.io/projected/1c3563f4-8c85-4267-acf0-020554ae7aa2-kube-api-access-7nrr5\") pod \"maas-api-revoke-sub-4b041902-qfqlp\" (UID: \"1c3563f4-8c85-4267-acf0-020554ae7aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"
Oct 29 22:04:13.854260 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:13.854218    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"
Oct 29 22:04:14.179176 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:04:14.179141    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3563f4_8c85_4267_acf0_020554ae7aa2.slice/crio-e30206284b28bf7a41bb2c0550df85252046b3315f733a1bbb57c3b6498c6d36 WatchSource:0}: Error finding container e30206284b28bf7a41bb2c0550df85252046b3315f733a1bbb57c3b6498c6d36: Status 404 returned error can't find the container with id e30206284b28bf7a41bb2c0550df85252046b3315f733a1bbb57c3b6498c6d36
Oct 29 22:04:14.182592 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:14.182562    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"]
Oct 29 22:04:15.143477 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:15.143439    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp" event={"ID":"1c3563f4-8c85-4267-acf0-020554ae7aa2","Type":"ContainerStarted","Data":"4d61e0996c289fac326d65d84ede8440a913f31bcd909aaada6a19455e87094d"}
Oct 29 22:04:15.143971 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:15.143492    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp" event={"ID":"1c3563f4-8c85-4267-acf0-020554ae7aa2","Type":"ContainerStarted","Data":"e30206284b28bf7a41bb2c0550df85252046b3315f733a1bbb57c3b6498c6d36"}
Oct 29 22:04:15.190322 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:15.190276    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp" podStartSLOduration=2.190262204 podStartE2EDuration="2.190262204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:15.184504666 +0000 UTC m=+1027.684546976" watchObservedRunningTime="2026-10-29 22:04:15.190262204 +0000 UTC m=+1027.690304502"
Oct 29 22:04:16.154991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:16.154951    7082 generic.go:356] "Generic (PLEG): container finished" podID="1c3563f4-8c85-4267-acf0-020554ae7aa2" containerID="4d61e0996c289fac326d65d84ede8440a913f31bcd909aaada6a19455e87094d" exitCode=0
Oct 29 22:04:16.155417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:16.155035    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp" event={"ID":"1c3563f4-8c85-4267-acf0-020554ae7aa2","Type":"ContainerDied","Data":"4d61e0996c289fac326d65d84ede8440a913f31bcd909aaada6a19455e87094d"}
Oct 29 22:04:17.389689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:17.386687    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"
Oct 29 22:04:17.438284 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:17.438202    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3563f4-8c85-4267-acf0-020554ae7aa2-kube-api-access-7nrr5\" (UniqueName: \"kubernetes.io/projected/1c3563f4-8c85-4267-acf0-020554ae7aa2-kube-api-access-7nrr5\") pod \"1c3563f4-8c85-4267-acf0-020554ae7aa2\" (UID: \"1c3563f4-8c85-4267-acf0-020554ae7aa2\") "
Oct 29 22:04:17.438533 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:17.438515    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-xks-ca\") pod \"1c3563f4-8c85-4267-acf0-020554ae7aa2\" (UID: \"1c3563f4-8c85-4267-acf0-020554ae7aa2\") "
Oct 29 22:04:17.438886 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:17.438667    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-openshift-ca\") pod \"1c3563f4-8c85-4267-acf0-020554ae7aa2\" (UID: \"1c3563f4-8c85-4267-acf0-020554ae7aa2\") "
Oct 29 22:04:17.439227 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:17.439202    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-openshift-ca" pod "1c3563f4-8c85-4267-acf0-020554ae7aa2" (UID: "1c3563f4-8c85-4267-acf0-020554ae7aa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:04:17.439466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:17.439444    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:04:17.441023 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:17.440991    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3563f4-8c85-4267-acf0-020554ae7aa2-kube-api-access-7nrr5" pod "1c3563f4-8c85-4267-acf0-020554ae7aa2" (UID: "1c3563f4-8c85-4267-acf0-020554ae7aa2"). InnerVolumeSpecName "kube-api-access-7nrr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:04:17.441132 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:17.441116    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-xks-ca" pod "1c3563f4-8c85-4267-acf0-020554ae7aa2" (UID: "1c3563f4-8c85-4267-acf0-020554ae7aa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:04:17.541102 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:17.541073    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nrr5\" (UniqueName: \"kubernetes.io/projected/1c3563f4-8c85-4267-acf0-020554ae7aa2-kube-api-access-7nrr5\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:04:17.541102 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:17.541102    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3563f4-8c85-4267-acf0-020554ae7aa2-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:04:18.175083 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:18.175055    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"
Oct 29 22:04:18.175253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:18.175085    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp" event={"ID":"1c3563f4-8c85-4267-acf0-020554ae7aa2","Type":"ContainerDied","Data":"e30206284b28bf7a41bb2c0550df85252046b3315f733a1bbb57c3b6498c6d36"}
Oct 29 22:04:18.175253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:18.175131    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e30206284b28bf7a41bb2c0550df85252046b3315f733a1bbb57c3b6498c6d36"
Oct 29 22:04:20.973069 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:20.973027    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"]
Oct 29 22:04:20.975076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:20.975042    7082 cpu_manager.go:414] "containerMap: removing container" podUID="1c3563f4-8c85-4267-acf0-020554ae7aa2" containerName="revoke-keys"
Oct 29 22:04:20.975076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:20.975072    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3563f4-8c85-4267-acf0-020554ae7aa2" containerName="revoke-keys"
Oct 29 22:04:20.975571 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:20.975555    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3563f4-8c85-4267-acf0-020554ae7aa2" containerName="revoke-keys"
Oct 29 22:04:20.990139 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:20.990111    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"
Oct 29 22:04:21.002581 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.002496    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"]
Oct 29 22:04:21.018025 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.017994    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:04:21.019011 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.018983    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:04:21.019252 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.019237    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:04:21.019496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.019480    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:04:21.205651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.119987    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08f293db-f5865\" (UID: \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"
Oct 29 22:04:21.205651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.120148    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59z9f\" (UniqueName: \"kubernetes.io/projected/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-kube-api-access-59z9f\") pod \"maas-api-revoke-sub-08f293db-f5865\" (UID: \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"
Oct 29 22:04:21.205651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.120316    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08f293db-f5865\" (UID: \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"
Oct 29 22:04:21.232740 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.224384    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08f293db-f5865\" (UID: \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"
Oct 29 22:04:21.232740 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.224531    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59z9f\" (UniqueName: \"kubernetes.io/projected/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-kube-api-access-59z9f\") pod \"maas-api-revoke-sub-08f293db-f5865\" (UID: \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"
Oct 29 22:04:21.232740 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.224716    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08f293db-f5865\" (UID: \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"
Oct 29 22:04:21.243020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.233574    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"
Oct 29 22:04:21.243020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.234621    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08f293db-f5865\" (UID: \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"
Oct 29 22:04:21.416416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.412386    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08f293db-f5865\" (UID: \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"
Oct 29 22:04:21.422284 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.422219    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59z9f\" (UniqueName: \"kubernetes.io/projected/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-kube-api-access-59z9f\") pod \"maas-api-revoke-sub-08f293db-f5865\" (UID: \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"
Oct 29 22:04:21.437003 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.436975    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"
Oct 29 22:04:21.462598 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.462564    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:04:21.626060 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:21.626023    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"
Oct 29 22:04:22.022689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:22.018929    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"]
Oct 29 22:04:22.031651 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:04:22.029787    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bd3607_1cfd_4d2d_b67e_1c0b933dc7e5.slice/crio-cda29943830cfa2ca5740f2f7a9ab1fb0c5bb738c5c67e42fdff6d83ab9b333c WatchSource:0}: Error finding container cda29943830cfa2ca5740f2f7a9ab1fb0c5bb738c5c67e42fdff6d83ab9b333c: Status 404 returned error can't find the container with id cda29943830cfa2ca5740f2f7a9ab1fb0c5bb738c5c67e42fdff6d83ab9b333c
Oct 29 22:04:22.678659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:22.663696    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865" event={"ID":"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5","Type":"ContainerStarted","Data":"cda29943830cfa2ca5740f2f7a9ab1fb0c5bb738c5c67e42fdff6d83ab9b333c"}
Oct 29 22:04:23.771653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:23.767889    7082 generic.go:356] "Generic (PLEG): container finished" podID="a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5" containerID="9ee41bc1d36c3df38853a8288e0831bba91c5dedc65efe2a737802a52a79c3a1" exitCode=0
Oct 29 22:04:23.771653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:23.768188    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865" event={"ID":"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5","Type":"ContainerDied","Data":"9ee41bc1d36c3df38853a8288e0831bba91c5dedc65efe2a737802a52a79c3a1"}
Oct 29 22:04:24.724647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:24.723738    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:24.741653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:24.726787    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": context deadline exceeded"
Oct 29 22:04:24.741653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:24.726873    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:04:24.857697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:24.857379    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.214773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:25.214718    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.224766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:25.224718    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.307779 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:25.307726    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.651655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:25.650754    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.803844 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:25.803784    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.828803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:25.828763    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:04:25.911247 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:25.890578    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:04:25.911247 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:25.901095    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:04:25.945679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:25.940912    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:04:25.945679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:25.941250    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:04:26.218829 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:26.218717    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.321857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:26.257190    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:04:26.521711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:26.263567    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:04:26.546569 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:26.336733    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.638048 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:26.638015    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:04:26.641781 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:26.641739    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:04:26.699560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:26.699508    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.721022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:26.701287    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:04:27.023569 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:26.491118    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:27.024215 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:27.024183    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.024336 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:26.944285    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.055677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:26.944339    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.080681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:26.849868    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.278390 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:27.278292    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:04:27.538001 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:27.537891    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:04:27.549641 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:27.549575    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.739652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:27.738386    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:04:27.857868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:27.857824    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.072656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:28.057717    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:04:28.072656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:28.066748    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.082198 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:28.082156    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.127870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:28.127776    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:28.146718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:28.141139    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.413785 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:28.354006    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:04:28.636659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:28.605925    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:04:28.703547 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:28.666457    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:04:29.223661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:29.131523    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.296651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:29.282335    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" podUID="0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.472831 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:29.472789    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:29.798656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:29.724802    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.798656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:29.519406    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.798656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:29.749122    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.798656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:29.724746    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:04:29.798656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:29.797547    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:29.798656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:29.797605    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.799304 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:04:29.507020    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:04:29.816653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:29.519436    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:04:29.866251 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:29.866209    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.014655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.013889    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded"
Oct 29 22:04:30.149648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:29.360776    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.175655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.171801    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.257649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.256984    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:04:30.285893 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.285849    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" podUID="0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.359653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.331299    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"
Oct 29 22:04:30.359653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.337701    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865" event={"ID":"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5","Type":"ContainerDied","Data":"cda29943830cfa2ca5740f2f7a9ab1fb0c5bb738c5c67e42fdff6d83ab9b333c"}
Oct 29 22:04:30.359653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.337769    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cda29943830cfa2ca5740f2f7a9ab1fb0c5bb738c5c67e42fdff6d83ab9b333c"
Oct 29 22:04:30.359653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.351301    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 22:04:30.359653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.351386    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"]
Oct 29 22:04:30.359653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.352888    7082 cpu_manager.go:414] "containerMap: removing container" podUID="a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5" containerName="revoke-keys"
Oct 29 22:04:30.359653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.352909    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5" containerName="revoke-keys"
Oct 29 22:04:30.364345 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.364319    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5" containerName="revoke-keys"
Oct 29 22:04:30.419096 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.410805    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-openshift-ca\") pod \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\" (UID: \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\") "
Oct 29 22:04:30.419096 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.410938    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-xks-ca\") pod \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\" (UID: \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\") "
Oct 29 22:04:30.419096 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.411024    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-kube-api-access-59z9f\" (UniqueName: \"kubernetes.io/projected/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-kube-api-access-59z9f\") pod \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\" (UID: \"a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5\") "
Oct 29 22:04:30.489401 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.489359    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-openshift-ca" pod "a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5" (UID: "a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:04:30.490527 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.490498    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:04:30.529346 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.529303    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:30.529684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.529589    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.530148 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.530125    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:30.530299 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.530279    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.559658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.532881    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"]
Oct 29 22:04:30.577801 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.577768    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p"
Oct 29 22:04:30.578018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.533808    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.610653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.607488    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f6171d-openshift-default-dockercfg-t2bql\""
Oct 29 22:04:30.659664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.654041    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-xks-ca" pod "a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5" (UID: "a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:04:30.739046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.730679    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.739046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.736697    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-envoy\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.739046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.736797    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-data\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.739046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.736982    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtkg5\" (UniqueName: \"kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-kube-api-access-qtkg5\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.739046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.737065    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-istio-token\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.739046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.737138    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-credential-socket\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.739046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.737220    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-certs\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.739046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.737298    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-socket\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.739046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.737542    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43619c62-7adb-49b0-94c8-870ec330cd19-istio-podinfo\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.739046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.737574    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43619c62-7adb-49b0-94c8-870ec330cd19-istiod-ca-cert\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.739046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.737737    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:04:30.763086 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.758803    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.763086 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.759444    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.831326 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.831285    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:04:30.848812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.848762    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-certs\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.849122 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.849098    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-socket\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.863787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.849435    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43619c62-7adb-49b0-94c8-870ec330cd19-istio-podinfo\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.868769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.864717    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-certs\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.868966 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.868943    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43619c62-7adb-49b0-94c8-870ec330cd19-istiod-ca-cert\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.872649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.869119    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-envoy\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.872649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.869178    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-data\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.872649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.869386    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtkg5\" (UniqueName: \"kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-kube-api-access-qtkg5\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.872649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.869461    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-istio-token\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.872649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.869523    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-credential-socket\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.872649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.868107    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.883651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.883602    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43619c62-7adb-49b0-94c8-870ec330cd19-istiod-ca-cert\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.895146 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.895117    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-credential-socket\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:30.987557 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.987473    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-kube-api-access-59z9f" pod "a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5" (UID: "a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5"). InnerVolumeSpecName "kube-api-access-59z9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:04:31.065654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.996686    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59z9f\" (UniqueName: \"kubernetes.io/projected/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5-kube-api-access-59z9f\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:04:31.228653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.195543    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-data\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:31.259786 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.232689    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.259786 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:30.865300    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-socket\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:31.349560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.267057    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43619c62-7adb-49b0-94c8-870ec330cd19-istio-podinfo\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:31.349560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.290695    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:04:31.436777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.436724    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:31.499801 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.499749    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.626672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.622593    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-envoy\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:31.626672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.623101    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-istio-token\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:31.626672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.623667    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtkg5\" (UniqueName: \"kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-kube-api-access-qtkg5\") pod \"e2e-api-a-f6171d-openshift-default-64847464f9-s44m2\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") " pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:31.767214 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.765990    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:04:31.770238 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.770199    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.770775 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.770747    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.832788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.832744    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.878830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.878724    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:04:32.126662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.899619    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:32.144960 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:31.964450    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"
Oct 29 22:04:32.236500 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:32.236459    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.251777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:32.251736    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Oct 29 22:04:32.351793 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:32.351744    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.783651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:32.783585    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.792659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:32.785757    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:04:32.808832 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:32.785865    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:04:32.859654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:32.844015    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:04:33.211662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:33.104713    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.343986 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:33.343943    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:04:33.371231 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:04:33.352257    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bd3607_1cfd_4d2d_b67e_1c0b933dc7e5.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:04:33.410932 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:33.352529    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.542653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:33.539758    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.854838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:33.831590    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:04:33.857020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:33.856981    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.861726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:33.861692    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:04:33.875742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:33.867819    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.106254 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.106149    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.399139 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.368490    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:04:34.445099 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.417429    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.472763 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.472711    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:34.495656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.485708    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.614733 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.613774    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:34.614733 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.613843    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.744233 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.737006    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.781202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.779806    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:34.781202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.779867    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.856810 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.856756    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.863340 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.861745    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.863340 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.862580    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.936944 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:34.936848    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.104916 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:35.067046    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.142735 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:35.122183    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:04:35.235224 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:35.209832    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"]
Oct 29 22:04:35.343727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:35.234931    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:04:35.344104 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:35.344054    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:04:35.344305 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:35.344278    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:04:35.384660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:35.375805    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.440653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:35.439726    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.616341 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:35.616302    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2" event={"ID":"43619c62-7adb-49b0-94c8-870ec330cd19","Type":"ContainerStarted","Data":"2d2a4a762461a75f26ccc59be5ed738353dc8132c6757f184fc1153683343506"}
Oct 29 22:04:35.952316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:35.952205    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.998656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:35.966747    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:04:35.998656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:35.969546    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:04:36.057913 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:36.057869    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.480653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:36.166898    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.480653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:36.455904    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:04:36.523657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:36.512744    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:37.021202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:36.953128    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:37.058656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:36.969029    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:37.199927 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:37.168443    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:37.465658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:37.463985    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:37.485595 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:37.485549    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:37.501652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:37.491242    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:04:37.905921 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:37.903485    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:37.927593 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:37.926964    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:37.927593 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:37.927047    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:38.045251 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.043280    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:38.045251 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.043345    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" podUID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:38.045251 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.043428    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:04:38.045251 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.044510    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:04:38.062656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.055236    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:38.184893 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.184783    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:38.297566 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.297525    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2" event={"ID":"43619c62-7adb-49b0-94c8-870ec330cd19","Type":"ContainerStarted","Data":"00d6c71b54c18baf5f3c04df568e9ac647fd849c24c10f71a5e19b2d308cac46"}
Oct 29 22:04:38.413417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.413350    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2" podStartSLOduration=11.413327555 podStartE2EDuration="11.413327555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:38.377436535 +0000 UTC m=+1050.877478836" watchObservedRunningTime="2026-10-29 22:04:38.413327555 +0000 UTC m=+1050.913369855"
Oct 29 22:04:38.466420 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.464773    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:38.493365 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.493320    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:04:38.502415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.496720    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:38.922657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.918507    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:38.928100 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.927743    7082 patch_prober.go:29] interesting pod/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Oct 29 22:04:38.928100 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:38.927802    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2" podUID="43619c62-7adb-49b0-94c8-870ec330cd19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Oct 29 22:04:39.053674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:39.047726    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:39.053674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:39.048157    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:04:39.065768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:39.065731    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:39.186659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:39.185685    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:04:39.444708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:39.443517    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:39.475662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:39.472746    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:04:39.502827 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:39.502778    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:39.529032 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:39.510914    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:04:39.917673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:39.917609    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded"
Oct 29 22:04:40.111653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.105800    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:40.111653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.107538    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:04:40.280185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.280072    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:40.442677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.437860    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:40.520307 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.520262    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:40.548408 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.548320    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:04:40.554111 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.554077    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:04:40.598187 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.589439    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:04:40.659158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.659110    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:04:40.659364 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.659194    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:04:40.659364 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.659252    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:04:40.663654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.663398    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:40.663654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.663504    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:04:40.666061 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.665927    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:04:40.712126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:40.711503    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:04:41.700394 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.700363    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"]
Oct 29 22:04:41.707774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.707497    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.754229 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.754033    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"]
Oct 29 22:04:41.755117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.755089    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.755427 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.755406    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.755527 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.755465    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfrv8\" (UniqueName: \"kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-kube-api-access-lfrv8\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.755619 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.755583    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.759945 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.759109    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.759945 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.759143    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.759945 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.759162    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.759945 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.759284    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.759945 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.759430    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.861270 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.861228    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.863068 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.863003    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfrv8\" (UniqueName: \"kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-kube-api-access-lfrv8\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.863422 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.863401    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.863769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.863733    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.863901 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.863883    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.864014 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.863999    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.864227 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.864208    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.864441 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.864422    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.864610 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.864593    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.866898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.866845    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.867334 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.867311    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.867765 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.867744    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.868916 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.868893    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.869308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.869281    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.895997 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.883394    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.911955 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.911914    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.941054 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.941010    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfrv8\" (UniqueName: \"kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-kube-api-access-lfrv8\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:41.945541 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:41.945514    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-pntbr\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:42.040029 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.039554    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:42.472728 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.472687    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"]
Oct 29 22:04:42.473133 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:04:42.473111    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a92f63_b5c0_4f00_9976_5bbc789b2bed.slice/crio-68a9d28d2863472363e495dcabe6b638ff5cad50283f9ded1c9ce7e3815f2893 WatchSource:0}: Error finding container 68a9d28d2863472363e495dcabe6b638ff5cad50283f9ded1c9ce7e3815f2893: Status 404 returned error can't find the container with id 68a9d28d2863472363e495dcabe6b638ff5cad50283f9ded1c9ce7e3815f2893
Oct 29 22:04:42.476138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.476100    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:04:42.476263 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.476165    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:04:42.476263 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.476211    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:04:42.641061 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.637004    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"]
Oct 29 22:04:42.641842 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.641817    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.682528 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.680379    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-380eea-openshift-default-dockercfg-lm8cf\""
Oct 29 22:04:42.684674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.684649    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b8ea59-e14b-4658-abea-a9234b68fbb9-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.685026 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.685005    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.685168 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.685153    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.685310 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.685295    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.685435 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.685420    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.685551 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.685536    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.685739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.685723    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cgzx\" (UniqueName: \"kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-kube-api-access-6cgzx\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.685998 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.685973    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.686121 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.686107    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.700579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.700549    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" event={"ID":"76a92f63-b5c0-4f00-9976-5bbc789b2bed","Type":"ContainerStarted","Data":"d82bec39cf4f34f2d7688641833b8b0044ac07fd0cfe0d375d516046a66ddf87"}
Oct 29 22:04:42.701049 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.701030    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" event={"ID":"76a92f63-b5c0-4f00-9976-5bbc789b2bed","Type":"ContainerStarted","Data":"68a9d28d2863472363e495dcabe6b638ff5cad50283f9ded1c9ce7e3815f2893"}
Oct 29 22:04:42.701760 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.701738    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"]
Oct 29 22:04:42.807789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.806067    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.807789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.806148    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.807789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.806346    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b8ea59-e14b-4658-abea-a9234b68fbb9-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.807789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.806563    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.807789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.806619    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.807789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.806690    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.807789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.806747    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.807789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.806791    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.807789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.806894    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgzx\" (UniqueName: \"kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-kube-api-access-6cgzx\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.900651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.871396    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.900651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.871649    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.900651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.871873    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.900651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.872950    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.900651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.900187    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.919651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.901287    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b8ea59-e14b-4658-abea-a9234b68fbb9-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.929174 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.929143    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.954471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.954406    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cgzx\" (UniqueName: \"kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-kube-api-access-6cgzx\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:42.955289 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:42.955261    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:43.005662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:43.000581    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" podStartSLOduration=2.000557889 podStartE2EDuration="2.000557889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:42.821406646 +0000 UTC m=+1055.321448945" watchObservedRunningTime="2026-10-29 22:04:43.000557889 +0000 UTC m=+1055.500600188"
Oct 29 22:04:43.012657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:43.006474    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:43.057653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:43.045289    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:43.060714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:43.059899    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:04:43.060714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:43.059960    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 29 22:04:43.929326 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:43.929297    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"]
Oct 29 22:04:44.002049 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:44.001707    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:04:44.002049 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:44.001803    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:04:44.002049 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:44.001846    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:04:44.041953 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:44.040998    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:04:44.041953 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:44.041052    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 29 22:04:44.954334 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:44.954293    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" event={"ID":"93b8ea59-e14b-4658-abea-a9234b68fbb9","Type":"ContainerStarted","Data":"867962ef794ba40d2870c2150b83fdac0911ec4e1caae3f4fac181e863f56027"}
Oct 29 22:04:44.954882 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:44.954747    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" event={"ID":"93b8ea59-e14b-4658-abea-a9234b68fbb9","Type":"ContainerStarted","Data":"2fb239a9bb45f4ec188e502770d21da8357692a7eb9af58368f12e1745350c98"}
Oct 29 22:04:45.022720 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:45.022685    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:45.030018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:45.029993    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:45.040698 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:45.040667    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:04:45.040931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:45.040905    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 29 22:04:45.125023 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:45.124958    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" podStartSLOduration=3.124936735 podStartE2EDuration="3.124936735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:45.053150633 +0000 UTC m=+1057.553192934" watchObservedRunningTime="2026-10-29 22:04:45.124936735 +0000 UTC m=+1057.624979035"
Oct 29 22:04:45.966918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:45.966883    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:45.967981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:45.967962    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:04:46.040889 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:46.040837    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:04:46.041083 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:46.040894    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 29 22:04:47.041376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:47.041341    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:04:47.042020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:47.041988    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 29 22:04:48.040619 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:48.040584    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:04:48.040835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:48.040672    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 29 22:04:49.043392 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:49.043356    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:04:49.044213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:49.044143    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 29 22:04:50.100656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:50.072769    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:04:50.100656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:50.072851    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 29 22:04:51.041085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:51.041047    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:04:51.041351 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:51.041320    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 29 22:04:52.045029 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:52.044998    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:52.045460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:52.045346    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:52.046064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:52.046047    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:04:57.054427 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:57.049006    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"]
Oct 29 22:04:57.084464 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:57.084429    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a9f9aa-sznv5"]
Oct 29 22:04:57.115821 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:57.115762    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:57.641611 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:57.641573    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af235450-f751-4529-9eb7-4493375ae287" path="/var/lib/kubelet/pods/af235450-f751-4529-9eb7-4493375ae287/volumes"
Oct 29 22:04:57.971663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:57.971182    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"]
Oct 29 22:04:57.976689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:04:57.976658    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"]
Oct 29 22:04:58.447529 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:04:58.447494    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:05:00.026836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:00.026800    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"]
Oct 29 22:05:00.027300 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:00.027046    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" podUID="3c28b204-2e05-473f-b027-13d086286e51" containerName="istio-proxy" containerID="cri-o://3ca400ba5a58244757b42cab9c51199345ab4cb6f5b471981cdfbe6810dff964" gracePeriod=30
Oct 29 22:05:02.971710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:02.971670    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5" podUID="2a691653-24d1-4772-813b-067bb8f78d97" containerName="payload-processing" containerID="cri-o://56ebedb5c9db1e76926b7def250cabe871aa2df0eadf3573603407f50a5f3b48" gracePeriod=25
Oct 29 22:05:02.976908 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:02.976877    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc" podUID="a0bfbbf9-27ba-414b-875d-ce9807964aa4" containerName="payload-pre-processing" containerID="cri-o://da47863bf5a3b9fe87b77e6e8d7028ecd068e216dcbcfec4dce69ee7e35231c5" gracePeriod=25
Oct 29 22:05:03.415055 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.415021    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"
Oct 29 22:05:03.461496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.461459    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"
Oct 29 22:05:03.553897 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.553861    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a691653-24d1-4772-813b-067bb8f78d97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a691653-24d1-4772-813b-067bb8f78d97-plugins-config-volume\") pod \"2a691653-24d1-4772-813b-067bb8f78d97\" (UID: \"2a691653-24d1-4772-813b-067bb8f78d97\") "
Oct 29 22:05:03.554077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.553935    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0bfbbf9-27ba-414b-875d-ce9807964aa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0bfbbf9-27ba-414b-875d-ce9807964aa4-plugins-config-volume\") pod \"a0bfbbf9-27ba-414b-875d-ce9807964aa4\" (UID: \"a0bfbbf9-27ba-414b-875d-ce9807964aa4\") "
Oct 29 22:05:03.554077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.554004    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a691653-24d1-4772-813b-067bb8f78d97-kube-api-access-grflk\" (UniqueName: \"kubernetes.io/projected/2a691653-24d1-4772-813b-067bb8f78d97-kube-api-access-grflk\") pod \"2a691653-24d1-4772-813b-067bb8f78d97\" (UID: \"2a691653-24d1-4772-813b-067bb8f78d97\") "
Oct 29 22:05:03.554266 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.554249    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0bfbbf9-27ba-414b-875d-ce9807964aa4-kube-api-access-m579s\" (UniqueName: \"kubernetes.io/projected/a0bfbbf9-27ba-414b-875d-ce9807964aa4-kube-api-access-m579s\") pod \"a0bfbbf9-27ba-414b-875d-ce9807964aa4\" (UID: \"a0bfbbf9-27ba-414b-875d-ce9807964aa4\") "
Oct 29 22:05:03.554869 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.554845    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0bfbbf9-27ba-414b-875d-ce9807964aa4-plugins-config-volume" pod "a0bfbbf9-27ba-414b-875d-ce9807964aa4" (UID: "a0bfbbf9-27ba-414b-875d-ce9807964aa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:05:03.555416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.555391    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a691653-24d1-4772-813b-067bb8f78d97-plugins-config-volume" pod "2a691653-24d1-4772-813b-067bb8f78d97" (UID: "2a691653-24d1-4772-813b-067bb8f78d97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:05:03.557961 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.557940    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bfbbf9-27ba-414b-875d-ce9807964aa4-kube-api-access-m579s" pod "a0bfbbf9-27ba-414b-875d-ce9807964aa4" (UID: "a0bfbbf9-27ba-414b-875d-ce9807964aa4"). InnerVolumeSpecName "kube-api-access-m579s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:05:03.560771 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.560740    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a691653-24d1-4772-813b-067bb8f78d97-kube-api-access-grflk" pod "2a691653-24d1-4772-813b-067bb8f78d97" (UID: "2a691653-24d1-4772-813b-067bb8f78d97"). InnerVolumeSpecName "kube-api-access-grflk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:05:03.657526 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.657497    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m579s\" (UniqueName: \"kubernetes.io/projected/a0bfbbf9-27ba-414b-875d-ce9807964aa4-kube-api-access-m579s\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:03.657526 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.657527    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a691653-24d1-4772-813b-067bb8f78d97-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:03.657741 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.657541    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0bfbbf9-27ba-414b-875d-ce9807964aa4-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:03.657741 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.657552    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grflk\" (UniqueName: \"kubernetes.io/projected/2a691653-24d1-4772-813b-067bb8f78d97-kube-api-access-grflk\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:03.848024 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.847944    7082 generic.go:356] "Generic (PLEG): container finished" podID="2a691653-24d1-4772-813b-067bb8f78d97" containerID="56ebedb5c9db1e76926b7def250cabe871aa2df0eadf3573603407f50a5f3b48" exitCode=0
Oct 29 22:05:03.848024 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.847990    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"
Oct 29 22:05:03.848228 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.848027    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5" event={"ID":"2a691653-24d1-4772-813b-067bb8f78d97","Type":"ContainerDied","Data":"56ebedb5c9db1e76926b7def250cabe871aa2df0eadf3573603407f50a5f3b48"}
Oct 29 22:05:03.848228 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.848093    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5" event={"ID":"2a691653-24d1-4772-813b-067bb8f78d97","Type":"ContainerDied","Data":"150aaba882ecd7a3d88bf845ec611f971aa854d4f417c272cd832a47c121ee3f"}
Oct 29 22:05:03.848228 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.848123    7082 scope.go:122] "RemoveContainer" containerID="56ebedb5c9db1e76926b7def250cabe871aa2df0eadf3573603407f50a5f3b48"
Oct 29 22:05:03.850022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.849771    7082 generic.go:356] "Generic (PLEG): container finished" podID="a0bfbbf9-27ba-414b-875d-ce9807964aa4" containerID="da47863bf5a3b9fe87b77e6e8d7028ecd068e216dcbcfec4dce69ee7e35231c5" exitCode=0
Oct 29 22:05:03.850022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.849846    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc" event={"ID":"a0bfbbf9-27ba-414b-875d-ce9807964aa4","Type":"ContainerDied","Data":"da47863bf5a3b9fe87b77e6e8d7028ecd068e216dcbcfec4dce69ee7e35231c5"}
Oct 29 22:05:03.850022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.849875    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"
Oct 29 22:05:03.850022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.849897    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc" event={"ID":"a0bfbbf9-27ba-414b-875d-ce9807964aa4","Type":"ContainerDied","Data":"ab86762080e27a5bdf54239f8361cdb7b61d335cd67ecc26ac424dcc26a9f5c7"}
Oct 29 22:05:03.860615 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.860597    7082 scope.go:122] "RemoveContainer" containerID="56ebedb5c9db1e76926b7def250cabe871aa2df0eadf3573603407f50a5f3b48"
Oct 29 22:05:03.863868 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:05:03.863830    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ebedb5c9db1e76926b7def250cabe871aa2df0eadf3573603407f50a5f3b48\": container with ID starting with 56ebedb5c9db1e76926b7def250cabe871aa2df0eadf3573603407f50a5f3b48 not found: ID does not exist" containerID="56ebedb5c9db1e76926b7def250cabe871aa2df0eadf3573603407f50a5f3b48"
Oct 29 22:05:03.863961 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.863881    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ebedb5c9db1e76926b7def250cabe871aa2df0eadf3573603407f50a5f3b48"} err="failed to get container status \"56ebedb5c9db1e76926b7def250cabe871aa2df0eadf3573603407f50a5f3b48\": rpc error: code = NotFound desc = could not find container \"56ebedb5c9db1e76926b7def250cabe871aa2df0eadf3573603407f50a5f3b48\": container with ID starting with 56ebedb5c9db1e76926b7def250cabe871aa2df0eadf3573603407f50a5f3b48 not found: ID does not exist"
Oct 29 22:05:03.863961 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.863904    7082 scope.go:122] "RemoveContainer" containerID="da47863bf5a3b9fe87b77e6e8d7028ecd068e216dcbcfec4dce69ee7e35231c5"
Oct 29 22:05:03.876377 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.876353    7082 scope.go:122] "RemoveContainer" containerID="da47863bf5a3b9fe87b77e6e8d7028ecd068e216dcbcfec4dce69ee7e35231c5"
Oct 29 22:05:03.876774 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:05:03.876748    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da47863bf5a3b9fe87b77e6e8d7028ecd068e216dcbcfec4dce69ee7e35231c5\": container with ID starting with da47863bf5a3b9fe87b77e6e8d7028ecd068e216dcbcfec4dce69ee7e35231c5 not found: ID does not exist" containerID="da47863bf5a3b9fe87b77e6e8d7028ecd068e216dcbcfec4dce69ee7e35231c5"
Oct 29 22:05:03.876858 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.876787    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da47863bf5a3b9fe87b77e6e8d7028ecd068e216dcbcfec4dce69ee7e35231c5"} err="failed to get container status \"da47863bf5a3b9fe87b77e6e8d7028ecd068e216dcbcfec4dce69ee7e35231c5\": rpc error: code = NotFound desc = could not find container \"da47863bf5a3b9fe87b77e6e8d7028ecd068e216dcbcfec4dce69ee7e35231c5\": container with ID starting with da47863bf5a3b9fe87b77e6e8d7028ecd068e216dcbcfec4dce69ee7e35231c5 not found: ID does not exist"
Oct 29 22:05:03.916676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.912000    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"]
Oct 29 22:05:03.922173 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.922135    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-6785f64979-f2fcc"]
Oct 29 22:05:03.954943 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.954909    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"]
Oct 29 22:05:03.964682 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:03.963002    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-734480-b8c46d65b-xc2x5"]
Oct 29 22:05:05.466375 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:05.465432    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"]
Oct 29 22:05:05.466375 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:05.465698    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg" podUID="9f82ffb2-04a4-4fff-916f-2b06335dd7bf" containerName="istio-proxy" containerID="cri-o://9f30503010576ed612a3ef94b254ddc93968b5efb94f25f50e2da8b6669115e5" gracePeriod=30
Oct 29 22:05:05.824316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:05.824284    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a691653-24d1-4772-813b-067bb8f78d97" path="/var/lib/kubelet/pods/2a691653-24d1-4772-813b-067bb8f78d97/volumes"
Oct 29 22:05:05.825285 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:05.825262    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0bfbbf9-27ba-414b-875d-ce9807964aa4" path="/var/lib/kubelet/pods/a0bfbbf9-27ba-414b-875d-ce9807964aa4/volumes"
Oct 29 22:05:06.038143 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.026786    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 22:05:06.092175 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.092087    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c28b204-2e05-473f-b027-13d086286e51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c28b204-2e05-473f-b027-13d086286e51-istiod-ca-cert\") pod \"3c28b204-2e05-473f-b027-13d086286e51\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") "
Oct 29 22:05:06.093523 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.093493    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-certs\") pod \"3c28b204-2e05-473f-b027-13d086286e51\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") "
Oct 29 22:05:06.093784 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.093768    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-data\") pod \"3c28b204-2e05-473f-b027-13d086286e51\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") "
Oct 29 22:05:06.093940 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.093927    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-socket\") pod \"3c28b204-2e05-473f-b027-13d086286e51\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") "
Oct 29 22:05:06.094066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.094050    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-envoy\") pod \"3c28b204-2e05-473f-b027-13d086286e51\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") "
Oct 29 22:05:06.094239 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.094224    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-kube-api-access-g87s2\" (UniqueName: \"kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-kube-api-access-g87s2\") pod \"3c28b204-2e05-473f-b027-13d086286e51\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") "
Oct 29 22:05:06.094387 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.094372    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-credential-socket\") pod \"3c28b204-2e05-473f-b027-13d086286e51\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") "
Oct 29 22:05:06.094529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.094514    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c28b204-2e05-473f-b027-13d086286e51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c28b204-2e05-473f-b027-13d086286e51-istio-podinfo\") pod \"3c28b204-2e05-473f-b027-13d086286e51\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") "
Oct 29 22:05:06.094721 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.094702    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-istio-token\" (UniqueName: \"kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-istio-token\") pod \"3c28b204-2e05-473f-b027-13d086286e51\" (UID: \"3c28b204-2e05-473f-b027-13d086286e51\") "
Oct 29 22:05:06.192754 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.187643    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-socket" pod "3c28b204-2e05-473f-b027-13d086286e51" (UID: "3c28b204-2e05-473f-b027-13d086286e51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:05:06.196273 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.193657    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-istio-token" pod "3c28b204-2e05-473f-b027-13d086286e51" (UID: "3c28b204-2e05-473f-b027-13d086286e51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:05:06.196273 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.194418    7082 generic.go:356] "Generic (PLEG): container finished" podID="3c28b204-2e05-473f-b027-13d086286e51" containerID="3ca400ba5a58244757b42cab9c51199345ab4cb6f5b471981cdfbe6810dff964" exitCode=0
Oct 29 22:05:06.196273 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.195110    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" event={"ID":"3c28b204-2e05-473f-b027-13d086286e51","Type":"ContainerDied","Data":"3ca400ba5a58244757b42cab9c51199345ab4cb6f5b471981cdfbe6810dff964"}
Oct 29 22:05:06.196273 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.195231    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s" event={"ID":"3c28b204-2e05-473f-b027-13d086286e51","Type":"ContainerDied","Data":"e58eca4c908c07d75ae18f36834145a83f054eba56b3df596686cf9413df6d83"}
Oct 29 22:05:06.197548 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.196842    7082 scope.go:122] "RemoveContainer" containerID="3ca400ba5a58244757b42cab9c51199345ab4cb6f5b471981cdfbe6810dff964"
Oct 29 22:05:06.198679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.198117    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"
Oct 29 22:05:06.226052 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.225982    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c28b204-2e05-473f-b027-13d086286e51-istiod-ca-cert" pod "3c28b204-2e05-473f-b027-13d086286e51" (UID: "3c28b204-2e05-473f-b027-13d086286e51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:05:06.337717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.315966    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.337937    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.338704    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c28b204-2e05-473f-b027-13d086286e51-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.339307    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-certs" pod "3c28b204-2e05-473f-b027-13d086286e51" (UID: "3c28b204-2e05-473f-b027-13d086286e51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.340394    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c28b204-2e05-473f-b027-13d086286e51-istio-podinfo" pod "3c28b204-2e05-473f-b027-13d086286e51" (UID: "3c28b204-2e05-473f-b027-13d086286e51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.342278    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-kube-api-access-g87s2" pod "3c28b204-2e05-473f-b027-13d086286e51" (UID: "3c28b204-2e05-473f-b027-13d086286e51"). InnerVolumeSpecName "kube-api-access-g87s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.345169    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-credential-socket" pod "3c28b204-2e05-473f-b027-13d086286e51" (UID: "3c28b204-2e05-473f-b027-13d086286e51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.347088    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-data" pod "3c28b204-2e05-473f-b027-13d086286e51" (UID: "3c28b204-2e05-473f-b027-13d086286e51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.347653    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-envoy" pod "3c28b204-2e05-473f-b027-13d086286e51" (UID: "3c28b204-2e05-473f-b027-13d086286e51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.347905    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"]
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.350051    7082 cpu_manager.go:414] "containerMap: removing container" podUID="3c28b204-2e05-473f-b027-13d086286e51" containerName="istio-proxy"
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.350076    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c28b204-2e05-473f-b027-13d086286e51" containerName="istio-proxy"
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.350094    7082 cpu_manager.go:414] "containerMap: removing container" podUID="3c28b204-2e05-473f-b027-13d086286e51" containerName="istio-proxy"
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.350101    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c28b204-2e05-473f-b027-13d086286e51" containerName="istio-proxy"
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.350118    7082 cpu_manager.go:414] "containerMap: removing container" podUID="2a691653-24d1-4772-813b-067bb8f78d97" containerName="payload-processing"
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.350126    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a691653-24d1-4772-813b-067bb8f78d97" containerName="payload-processing"
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.350168    7082 cpu_manager.go:414] "containerMap: removing container" podUID="a0bfbbf9-27ba-414b-875d-ce9807964aa4" containerName="payload-pre-processing"
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.350176    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0bfbbf9-27ba-414b-875d-ce9807964aa4" containerName="payload-pre-processing"
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.350389    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c28b204-2e05-473f-b027-13d086286e51" containerName="istio-proxy"
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.350401    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c28b204-2e05-473f-b027-13d086286e51" containerName="istio-proxy"
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.350410    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a691653-24d1-4772-813b-067bb8f78d97" containerName="payload-processing"
Oct 29 22:05:06.350522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.350427    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0bfbbf9-27ba-414b-875d-ce9807964aa4" containerName="payload-pre-processing"
Oct 29 22:05:06.358112 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.358081    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"
Oct 29 22:05:06.359861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.359837    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"]
Oct 29 22:05:06.366090 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.366069    7082 scope.go:122] "RemoveContainer" containerID="f1ff411dd64f982b0603fb68bd3230ba10852f736d1c301d91ebe48d946a438d"
Oct 29 22:05:06.374673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.374651    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:05:06.380337 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.380314    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:05:06.380758 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.380737    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:05:06.380947 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.380930    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:05:06.384748 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.384403    7082 scope.go:122] "RemoveContainer" containerID="3ca400ba5a58244757b42cab9c51199345ab4cb6f5b471981cdfbe6810dff964"
Oct 29 22:05:06.385084 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:05:06.384935    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca400ba5a58244757b42cab9c51199345ab4cb6f5b471981cdfbe6810dff964\": container with ID starting with 3ca400ba5a58244757b42cab9c51199345ab4cb6f5b471981cdfbe6810dff964 not found: ID does not exist" containerID="3ca400ba5a58244757b42cab9c51199345ab4cb6f5b471981cdfbe6810dff964"
Oct 29 22:05:06.385084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.384973    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca400ba5a58244757b42cab9c51199345ab4cb6f5b471981cdfbe6810dff964"} err="failed to get container status \"3ca400ba5a58244757b42cab9c51199345ab4cb6f5b471981cdfbe6810dff964\": rpc error: code = NotFound desc = could not find container \"3ca400ba5a58244757b42cab9c51199345ab4cb6f5b471981cdfbe6810dff964\": container with ID starting with 3ca400ba5a58244757b42cab9c51199345ab4cb6f5b471981cdfbe6810dff964 not found: ID does not exist"
Oct 29 22:05:06.385084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.384997    7082 scope.go:122] "RemoveContainer" containerID="f1ff411dd64f982b0603fb68bd3230ba10852f736d1c301d91ebe48d946a438d"
Oct 29 22:05:06.385435 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:05:06.385367    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ff411dd64f982b0603fb68bd3230ba10852f736d1c301d91ebe48d946a438d\": container with ID starting with f1ff411dd64f982b0603fb68bd3230ba10852f736d1c301d91ebe48d946a438d not found: ID does not exist" containerID="f1ff411dd64f982b0603fb68bd3230ba10852f736d1c301d91ebe48d946a438d"
Oct 29 22:05:06.385435 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.385398    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ff411dd64f982b0603fb68bd3230ba10852f736d1c301d91ebe48d946a438d"} err="failed to get container status \"f1ff411dd64f982b0603fb68bd3230ba10852f736d1c301d91ebe48d946a438d\": rpc error: code = NotFound desc = could not find container \"f1ff411dd64f982b0603fb68bd3230ba10852f736d1c301d91ebe48d946a438d\": container with ID starting with f1ff411dd64f982b0603fb68bd3230ba10852f736d1c301d91ebe48d946a438d not found: ID does not exist"
Oct 29 22:05:06.441225 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.441182    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6df539-9p8bm\" (UID: \"15da3721-b9f1-4140-b963-6d08b418814c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"
Oct 29 22:05:06.441487 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.441353    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6df539-9p8bm\" (UID: \"15da3721-b9f1-4140-b963-6d08b418814c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"
Oct 29 22:05:06.441487 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.441422    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmhnr\" (UniqueName: \"kubernetes.io/projected/15da3721-b9f1-4140-b963-6d08b418814c-kube-api-access-rmhnr\") pod \"maas-api-revoke-keys-e2e-api-b-6df539-9p8bm\" (UID: \"15da3721-b9f1-4140-b963-6d08b418814c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"
Oct 29 22:05:06.441693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.441579    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:06.441693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.441596    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:06.441693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.441604    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:06.441693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.441615    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g87s2\" (UniqueName: \"kubernetes.io/projected/3c28b204-2e05-473f-b027-13d086286e51-kube-api-access-g87s2\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:06.441693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.441646    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c28b204-2e05-473f-b027-13d086286e51-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:06.441693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.441659    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c28b204-2e05-473f-b027-13d086286e51-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:06.543521 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.543484    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6df539-9p8bm\" (UID: \"15da3721-b9f1-4140-b963-6d08b418814c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"
Oct 29 22:05:06.544067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.543576    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6df539-9p8bm\" (UID: \"15da3721-b9f1-4140-b963-6d08b418814c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"
Oct 29 22:05:06.544067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.543649    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmhnr\" (UniqueName: \"kubernetes.io/projected/15da3721-b9f1-4140-b963-6d08b418814c-kube-api-access-rmhnr\") pod \"maas-api-revoke-keys-e2e-api-b-6df539-9p8bm\" (UID: \"15da3721-b9f1-4140-b963-6d08b418814c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"
Oct 29 22:05:06.544799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.544765    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6df539-9p8bm\" (UID: \"15da3721-b9f1-4140-b963-6d08b418814c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"
Oct 29 22:05:06.546447 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.546421    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6df539-9p8bm\" (UID: \"15da3721-b9f1-4140-b963-6d08b418814c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"
Oct 29 22:05:06.556397 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.556373    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"]
Oct 29 22:05:06.563753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.563729    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-f556s"]
Oct 29 22:05:06.569390 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.569366    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmhnr\" (UniqueName: \"kubernetes.io/projected/15da3721-b9f1-4140-b963-6d08b418814c-kube-api-access-rmhnr\") pod \"maas-api-revoke-keys-e2e-api-b-6df539-9p8bm\" (UID: \"15da3721-b9f1-4140-b963-6d08b418814c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"
Oct 29 22:05:06.685203 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:06.685077    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"
Oct 29 22:05:07.043510 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:05:07.043478    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15da3721_b9f1_4140_b963_6d08b418814c.slice/crio-b6a61c927eaec784dbc2ca51daa2c2fca1df44da6043efeced47cece213ba6db WatchSource:0}: Error finding container b6a61c927eaec784dbc2ca51daa2c2fca1df44da6043efeced47cece213ba6db: Status 404 returned error can't find the container with id b6a61c927eaec784dbc2ca51daa2c2fca1df44da6043efeced47cece213ba6db
Oct 29 22:05:07.044770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:07.044744    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"]
Oct 29 22:05:07.211357 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:07.211320    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm" event={"ID":"15da3721-b9f1-4140-b963-6d08b418814c","Type":"ContainerStarted","Data":"b6a61c927eaec784dbc2ca51daa2c2fca1df44da6043efeced47cece213ba6db"}
Oct 29 22:05:07.685333 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:07.685291    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c28b204-2e05-473f-b027-13d086286e51" path="/var/lib/kubelet/pods/3c28b204-2e05-473f-b027-13d086286e51/volumes"
Oct 29 22:05:08.234451 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:08.234411    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm" event={"ID":"15da3721-b9f1-4140-b963-6d08b418814c","Type":"ContainerStarted","Data":"821bce1497ebe7c190a7a37a145e84d9b3c25e2bb9b3112f77f5f1991956201d"}
Oct 29 22:05:08.280862 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:08.280810    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm" podStartSLOduration=2.280797101 podStartE2EDuration="2.280797101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:08.275142635 +0000 UTC m=+1080.775184934" watchObservedRunningTime="2026-10-29 22:05:08.280797101 +0000 UTC m=+1080.780839400"
Oct 29 22:05:08.938237 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:08.938199    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"]
Oct 29 22:05:08.969604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:08.967596    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:08.974613 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:08.974580    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"]
Oct 29 22:05:08.987672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:08.987589    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-dd38c2-openshift-default-dockercfg-bqcds\""
Oct 29 22:05:09.304149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.300918    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.304149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.301026    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.304149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.301065    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.304149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.301370    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.304149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.301399    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7332e1-ce42-4071-bf1a-ddf706996684-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.304149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.301431    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt2n8\" (UniqueName: \"kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-kube-api-access-tt2n8\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.304149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.301514    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.304149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.301619    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.304149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.301724    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.365131 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.365082    7082 scope.go:122] "RemoveContainer" containerID="57691b086424e890e296c68505e5cd090b3027d5a8b3b66fcbb2643dc919174b"
Oct 29 22:05:09.370184 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.370147    7082 generic.go:356] "Generic (PLEG): container finished" podID="15da3721-b9f1-4140-b963-6d08b418814c" containerID="821bce1497ebe7c190a7a37a145e84d9b3c25e2bb9b3112f77f5f1991956201d" exitCode=0
Oct 29 22:05:09.370663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.370613    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm" event={"ID":"15da3721-b9f1-4140-b963-6d08b418814c","Type":"ContainerDied","Data":"821bce1497ebe7c190a7a37a145e84d9b3c25e2bb9b3112f77f5f1991956201d"}
Oct 29 22:05:09.524669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.524404    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.524669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.524501    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.524669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.524563    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.537148 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.537100    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.538865 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.538830    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.539157 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.539138    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.539593 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.539573    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.539743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.539728    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7332e1-ce42-4071-bf1a-ddf706996684-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.539906 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.539835    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt2n8\" (UniqueName: \"kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-kube-api-access-tt2n8\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.541467 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.541443    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.544375 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.544354    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.577925 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.573399    7082 scope.go:122] "RemoveContainer" containerID="295a2746ba7f6a52c281a38ba71b26758e4231249eaa99be26bfc1c720dd6389"
Oct 29 22:05:09.655281 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.654689    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.655281 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.654952    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.664431 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.662939    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7332e1-ce42-4071-bf1a-ddf706996684-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.673240 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.673107    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.674492 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.673970    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt2n8\" (UniqueName: \"kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-kube-api-access-tt2n8\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.675499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.675442    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.675835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.675813    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:09.903044 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:09.903014    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:10.631536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.486149    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"]
Oct 29 22:05:10.631536 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:05:10.486683    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7332e1_ce42_4071_bf1a_ddf706996684.slice/crio-2bbf80ae0ad0112fd0cd6e89aca45592d7257172af1122068fe30d687201047d WatchSource:0}: Error finding container 2bbf80ae0ad0112fd0cd6e89aca45592d7257172af1122068fe30d687201047d: Status 404 returned error can't find the container with id 2bbf80ae0ad0112fd0cd6e89aca45592d7257172af1122068fe30d687201047d
Oct 29 22:05:10.631536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.490492    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:05:10.631536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.490992    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:05:10.631536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.491059    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:05:10.631536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.568783    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"
Oct 29 22:05:10.715339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.715302    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-xks-ca\") pod \"15da3721-b9f1-4140-b963-6d08b418814c\" (UID: \"15da3721-b9f1-4140-b963-6d08b418814c\") "
Oct 29 22:05:10.715528 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.715378    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-openshift-ca\") pod \"15da3721-b9f1-4140-b963-6d08b418814c\" (UID: \"15da3721-b9f1-4140-b963-6d08b418814c\") "
Oct 29 22:05:10.715621 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.715600    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15da3721-b9f1-4140-b963-6d08b418814c-kube-api-access-rmhnr\" (UniqueName: \"kubernetes.io/projected/15da3721-b9f1-4140-b963-6d08b418814c-kube-api-access-rmhnr\") pod \"15da3721-b9f1-4140-b963-6d08b418814c\" (UID: \"15da3721-b9f1-4140-b963-6d08b418814c\") "
Oct 29 22:05:10.716241 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.716215    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-openshift-ca" pod "15da3721-b9f1-4140-b963-6d08b418814c" (UID: "15da3721-b9f1-4140-b963-6d08b418814c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:05:10.717331 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.717306    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:10.719176 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.719149    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15da3721-b9f1-4140-b963-6d08b418814c-kube-api-access-rmhnr" pod "15da3721-b9f1-4140-b963-6d08b418814c" (UID: "15da3721-b9f1-4140-b963-6d08b418814c"). InnerVolumeSpecName "kube-api-access-rmhnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:05:10.723909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.723877    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-xks-ca" pod "15da3721-b9f1-4140-b963-6d08b418814c" (UID: "15da3721-b9f1-4140-b963-6d08b418814c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:05:10.819336 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.819305    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmhnr\" (UniqueName: \"kubernetes.io/projected/15da3721-b9f1-4140-b963-6d08b418814c-kube-api-access-rmhnr\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:10.819336 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:10.819337    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15da3721-b9f1-4140-b963-6d08b418814c-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:11.155585 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.155560    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:05:11.243706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.243400    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-podinfo\") pod \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") "
Oct 29 22:05:11.243706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.243517    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-socket\") pod \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") "
Oct 29 22:05:11.243706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.243702    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istiod-ca-cert\") pod \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") "
Oct 29 22:05:11.243992 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.243757    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-certs\") pod \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") "
Oct 29 22:05:11.243992 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.243844    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-kube-api-access-djdhd\" (UniqueName: \"kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-kube-api-access-djdhd\") pod \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") "
Oct 29 22:05:11.243992 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.243894    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-token\" (UniqueName: \"kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-token\") pod \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") "
Oct 29 22:05:11.243992 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.243973    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-credential-socket\") pod \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") "
Oct 29 22:05:11.244209 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.244068    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-data\") pod \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") "
Oct 29 22:05:11.244209 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.244137    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-envoy\") pod \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\" (UID: \"9f82ffb2-04a4-4fff-916f-2b06335dd7bf\") "
Oct 29 22:05:11.272615 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.268093    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-socket" pod "9f82ffb2-04a4-4fff-916f-2b06335dd7bf" (UID: "9f82ffb2-04a4-4fff-916f-2b06335dd7bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:05:11.272615 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.268374    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-data" pod "9f82ffb2-04a4-4fff-916f-2b06335dd7bf" (UID: "9f82ffb2-04a4-4fff-916f-2b06335dd7bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:05:11.272615 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.268680    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istiod-ca-cert" pod "9f82ffb2-04a4-4fff-916f-2b06335dd7bf" (UID: "9f82ffb2-04a4-4fff-916f-2b06335dd7bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:05:11.280996 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.280733    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-credential-socket" pod "9f82ffb2-04a4-4fff-916f-2b06335dd7bf" (UID: "9f82ffb2-04a4-4fff-916f-2b06335dd7bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:05:11.282452 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.282409    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-podinfo" pod "9f82ffb2-04a4-4fff-916f-2b06335dd7bf" (UID: "9f82ffb2-04a4-4fff-916f-2b06335dd7bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:05:11.282859 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.282815    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-token" pod "9f82ffb2-04a4-4fff-916f-2b06335dd7bf" (UID: "9f82ffb2-04a4-4fff-916f-2b06335dd7bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:05:11.286298 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.286262    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-envoy" pod "9f82ffb2-04a4-4fff-916f-2b06335dd7bf" (UID: "9f82ffb2-04a4-4fff-916f-2b06335dd7bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:05:11.292666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.289714    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-certs" pod "9f82ffb2-04a4-4fff-916f-2b06335dd7bf" (UID: "9f82ffb2-04a4-4fff-916f-2b06335dd7bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:05:11.292666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.290550    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:11.292666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.290577    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:11.292666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.290590    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:11.292666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.290602    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:11.292666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.290618    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:11.292666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.290659    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:11.292666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.290672    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:11.292666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.290686    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:11.314592 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.314537    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-kube-api-access-djdhd" pod "9f82ffb2-04a4-4fff-916f-2b06335dd7bf" (UID: "9f82ffb2-04a4-4fff-916f-2b06335dd7bf"). InnerVolumeSpecName "kube-api-access-djdhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:05:11.430743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.427944    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djdhd\" (UniqueName: \"kubernetes.io/projected/9f82ffb2-04a4-4fff-916f-2b06335dd7bf-kube-api-access-djdhd\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:11.502726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.502294    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" event={"ID":"5f7332e1-ce42-4071-bf1a-ddf706996684","Type":"ContainerStarted","Data":"a8703b676a08da768e9ed481426b8ca9ef2e64ac4db335f3e4d562eacf708558"}
Oct 29 22:05:11.502726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.502355    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" event={"ID":"5f7332e1-ce42-4071-bf1a-ddf706996684","Type":"ContainerStarted","Data":"2bbf80ae0ad0112fd0cd6e89aca45592d7257172af1122068fe30d687201047d"}
Oct 29 22:05:11.503733 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.503712    7082 generic.go:356] "Generic (PLEG): container finished" podID="9f82ffb2-04a4-4fff-916f-2b06335dd7bf" containerID="9f30503010576ed612a3ef94b254ddc93968b5efb94f25f50e2da8b6669115e5" exitCode=0
Oct 29 22:05:11.503932 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.503915    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"
Oct 29 22:05:11.504096 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.503872    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg" event={"ID":"9f82ffb2-04a4-4fff-916f-2b06335dd7bf","Type":"ContainerDied","Data":"9f30503010576ed612a3ef94b254ddc93968b5efb94f25f50e2da8b6669115e5"}
Oct 29 22:05:11.504256 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.504243    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg" event={"ID":"9f82ffb2-04a4-4fff-916f-2b06335dd7bf","Type":"ContainerDied","Data":"ecddc087ef42b8efb5b1d1983bf693449e6df606a05797da9f7686aceaf6e5cb"}
Oct 29 22:05:11.504383 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.504345    7082 scope.go:122] "RemoveContainer" containerID="9f30503010576ed612a3ef94b254ddc93968b5efb94f25f50e2da8b6669115e5"
Oct 29 22:05:11.507743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.507255    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm" event={"ID":"15da3721-b9f1-4140-b963-6d08b418814c","Type":"ContainerDied","Data":"b6a61c927eaec784dbc2ca51daa2c2fca1df44da6043efeced47cece213ba6db"}
Oct 29 22:05:11.507743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.507298    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6a61c927eaec784dbc2ca51daa2c2fca1df44da6043efeced47cece213ba6db"
Oct 29 22:05:11.507743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.507358    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"
Oct 29 22:05:11.523822 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.523799    7082 scope.go:122] "RemoveContainer" containerID="9f30503010576ed612a3ef94b254ddc93968b5efb94f25f50e2da8b6669115e5"
Oct 29 22:05:11.524172 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:05:11.524149    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f30503010576ed612a3ef94b254ddc93968b5efb94f25f50e2da8b6669115e5\": container with ID starting with 9f30503010576ed612a3ef94b254ddc93968b5efb94f25f50e2da8b6669115e5 not found: ID does not exist" containerID="9f30503010576ed612a3ef94b254ddc93968b5efb94f25f50e2da8b6669115e5"
Oct 29 22:05:11.524222 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.524185    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f30503010576ed612a3ef94b254ddc93968b5efb94f25f50e2da8b6669115e5"} err="failed to get container status \"9f30503010576ed612a3ef94b254ddc93968b5efb94f25f50e2da8b6669115e5\": rpc error: code = NotFound desc = could not find container \"9f30503010576ed612a3ef94b254ddc93968b5efb94f25f50e2da8b6669115e5\": container with ID starting with 9f30503010576ed612a3ef94b254ddc93968b5efb94f25f50e2da8b6669115e5 not found: ID does not exist"
Oct 29 22:05:11.617795 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.617686    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" podStartSLOduration=3.6176663639999997 podStartE2EDuration="3.617666364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:11.561449615 +0000 UTC m=+1084.061491914" watchObservedRunningTime="2026-10-29 22:05:11.617666364 +0000 UTC m=+1084.117708664"
Oct 29 22:05:11.620368 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.620340    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"]
Oct 29 22:05:11.635561 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.635528    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a9f9aa-openshift-default-7fd8548c55-6c7lg"]
Oct 29 22:05:11.903742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.903706    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:11.909479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:11.909452    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:12.532388 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:12.532344    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:12.533781 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:12.533755    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:05:13.633579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:13.633540    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f82ffb2-04a4-4fff-916f-2b06335dd7bf" path="/var/lib/kubelet/pods/9f82ffb2-04a4-4fff-916f-2b06335dd7bf/volumes"
Oct 29 22:05:15.828597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:15.828563    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"]
Oct 29 22:05:15.829393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:15.829376    7082 cpu_manager.go:414] "containerMap: removing container" podUID="15da3721-b9f1-4140-b963-6d08b418814c" containerName="revoke-keys"
Oct 29 22:05:15.829393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:15.829393    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15da3721-b9f1-4140-b963-6d08b418814c" containerName="revoke-keys"
Oct 29 22:05:15.829487 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:15.829446    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9f82ffb2-04a4-4fff-916f-2b06335dd7bf" containerName="istio-proxy"
Oct 29 22:05:15.829487 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:15.829451    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f82ffb2-04a4-4fff-916f-2b06335dd7bf" containerName="istio-proxy"
Oct 29 22:05:15.829591 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:15.829578    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f82ffb2-04a4-4fff-916f-2b06335dd7bf" containerName="istio-proxy"
Oct 29 22:05:15.829701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:15.829598    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="15da3721-b9f1-4140-b963-6d08b418814c" containerName="revoke-keys"
Oct 29 22:05:15.835493 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:15.835474    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"
Oct 29 22:05:15.853557 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:15.853535    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"]
Oct 29 22:05:15.857603 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:15.857583    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-dd38c2-dockercfg-gzw7d\""
Oct 29 22:05:15.857603 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:15.857587    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-dd38c2\""
Oct 29 22:05:15.894947 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:15.894916    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q58wm\" (UniqueName: \"kubernetes.io/projected/462635bd-9dd1-4132-b1a4-5eba418881f1-kube-api-access-q58wm\") pod \"payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z\" (UID: \"462635bd-9dd1-4132-b1a4-5eba418881f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"
Oct 29 22:05:15.895201 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:15.895180    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462635bd-9dd1-4132-b1a4-5eba418881f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z\" (UID: \"462635bd-9dd1-4132-b1a4-5eba418881f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"
Oct 29 22:05:16.028654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.023057    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58wm\" (UniqueName: \"kubernetes.io/projected/462635bd-9dd1-4132-b1a4-5eba418881f1-kube-api-access-q58wm\") pod \"payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z\" (UID: \"462635bd-9dd1-4132-b1a4-5eba418881f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"
Oct 29 22:05:16.028654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.023118    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462635bd-9dd1-4132-b1a4-5eba418881f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z\" (UID: \"462635bd-9dd1-4132-b1a4-5eba418881f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"
Oct 29 22:05:16.028654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.024743    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462635bd-9dd1-4132-b1a4-5eba418881f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z\" (UID: \"462635bd-9dd1-4132-b1a4-5eba418881f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"
Oct 29 22:05:16.072976 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.036271    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"]
Oct 29 22:05:16.238465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.238278    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q58wm\" (UniqueName: \"kubernetes.io/projected/462635bd-9dd1-4132-b1a4-5eba418881f1-kube-api-access-q58wm\") pod \"payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z\" (UID: \"462635bd-9dd1-4132-b1a4-5eba418881f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"
Oct 29 22:05:16.280736 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.270087    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"
Oct 29 22:05:16.331651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.331024    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"]
Oct 29 22:05:16.459652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.378478    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9\" (UID: \"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"
Oct 29 22:05:16.459652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.378895    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzvw9\" (UniqueName: \"kubernetes.io/projected/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-kube-api-access-lzvw9\") pod \"payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9\" (UID: \"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"
Oct 29 22:05:16.459652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.452035    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"
Oct 29 22:05:16.513445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.513358    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9\" (UID: \"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"
Oct 29 22:05:16.514173 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.514122    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzvw9\" (UniqueName: \"kubernetes.io/projected/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-kube-api-access-lzvw9\") pod \"payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9\" (UID: \"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"
Oct 29 22:05:16.515894 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.515807    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9\" (UID: \"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"
Oct 29 22:05:16.585135 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.585104    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzvw9\" (UniqueName: \"kubernetes.io/projected/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-kube-api-access-lzvw9\") pod \"payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9\" (UID: \"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"
Oct 29 22:05:16.592689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.592660    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"]
Oct 29 22:05:16.654965 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:16.654938    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"
Oct 29 22:05:17.040460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:17.040424    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"]
Oct 29 22:05:17.045650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:17.045602    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a65e410-qj4vg"]
Oct 29 22:05:17.245551 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:17.245465    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"]
Oct 29 22:05:17.301812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:17.293004    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"]
Oct 29 22:05:17.637857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:17.637830    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9031e15e-d7af-479a-8eb5-0b985f2fae99" path="/var/lib/kubelet/pods/9031e15e-d7af-479a-8eb5-0b985f2fae99/volumes"
Oct 29 22:05:17.663561 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:17.663525    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z" event={"ID":"462635bd-9dd1-4132-b1a4-5eba418881f1","Type":"ContainerStarted","Data":"6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9"}
Oct 29 22:05:17.663728 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:17.663585    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z" event={"ID":"462635bd-9dd1-4132-b1a4-5eba418881f1","Type":"ContainerStarted","Data":"afa87a6c01f47cb0260a40f1a9d83e13dd7cf5557fc30529b22790615692f482"}
Oct 29 22:05:17.663728 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:17.663612    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"
Oct 29 22:05:17.665412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:17.665390    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9" event={"ID":"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d","Type":"ContainerStarted","Data":"b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0"}
Oct 29 22:05:17.665522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:17.665428    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9" event={"ID":"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d","Type":"ContainerStarted","Data":"e3dbddd24593257226bec4da9e87f6d7b0b2802ecc705ccca8331c44cd58548b"}
Oct 29 22:05:17.665522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:17.665477    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"
Oct 29 22:05:17.709930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:17.709879    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z" podStartSLOduration=2.709866397 podStartE2EDuration="2.709866397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:17.705726352 +0000 UTC m=+1090.205768651" watchObservedRunningTime="2026-10-29 22:05:17.709866397 +0000 UTC m=+1090.209908687"
Oct 29 22:05:17.766568 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:17.766511    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9" podStartSLOduration=2.766496875 podStartE2EDuration="2.766496875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:17.746392063 +0000 UTC m=+1090.246434365" watchObservedRunningTime="2026-10-29 22:05:17.766496875 +0000 UTC m=+1090.266539174"
Oct 29 22:05:21.460666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:21.459808    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"]
Oct 29 22:05:21.496741 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:21.478254    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"
Oct 29 22:05:21.520652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:21.509927    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4\" (UID: \"17f87fff-d6eb-4aa9-b6b5-cac68c2fed28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"
Oct 29 22:05:21.520652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:21.510297    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrsj2\" (UniqueName: \"kubernetes.io/projected/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-kube-api-access-qrsj2\") pod \"payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4\" (UID: \"17f87fff-d6eb-4aa9-b6b5-cac68c2fed28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"
Oct 29 22:05:21.532651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:21.528171    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-734480\""
Oct 29 22:05:21.532651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:21.528407    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-734480-dockercfg-kwhjs\""
Oct 29 22:05:21.668714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:21.627881    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrsj2\" (UniqueName: \"kubernetes.io/projected/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-kube-api-access-qrsj2\") pod \"payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4\" (UID: \"17f87fff-d6eb-4aa9-b6b5-cac68c2fed28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"
Oct 29 22:05:21.668714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:21.628066    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4\" (UID: \"17f87fff-d6eb-4aa9-b6b5-cac68c2fed28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"
Oct 29 22:05:21.668714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:21.629376    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4\" (UID: \"17f87fff-d6eb-4aa9-b6b5-cac68c2fed28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"
Oct 29 22:05:21.668714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:21.649946    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz" podUID="26bfc7bc-ef5e-4df2-998b-bdc774f39f89" containerName="payload-processing" containerID="cri-o://62e3f8b61fe46b3b51fd2cb1a2bc7e91ab091fce77b94ea5bf9ea7cff9abb4f5" gracePeriod=25
Oct 29 22:05:21.882717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:21.787944    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrsj2\" (UniqueName: \"kubernetes.io/projected/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-kube-api-access-qrsj2\") pod \"payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4\" (UID: \"17f87fff-d6eb-4aa9-b6b5-cac68c2fed28\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"
Oct 29 22:05:21.916138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:21.914703    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"
Oct 29 22:05:22.286777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.286691    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"]
Oct 29 22:05:22.287059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.287036    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"]
Oct 29 22:05:22.316705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.315182    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"]
Oct 29 22:05:22.343299 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.328663    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"
Oct 29 22:05:22.373019 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.372929    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng7cx\" (UniqueName: \"kubernetes.io/projected/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-kube-api-access-ng7cx\") pod \"payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4\" (UID: \"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"
Oct 29 22:05:22.373178 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.373131    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4\" (UID: \"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"
Oct 29 22:05:22.475360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.475320    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng7cx\" (UniqueName: \"kubernetes.io/projected/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-kube-api-access-ng7cx\") pod \"payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4\" (UID: \"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"
Oct 29 22:05:22.475750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.475513    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4\" (UID: \"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"
Oct 29 22:05:22.476477 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.476447    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4\" (UID: \"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"
Oct 29 22:05:22.502719 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.502689    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng7cx\" (UniqueName: \"kubernetes.io/projected/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-kube-api-access-ng7cx\") pod \"payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4\" (UID: \"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"
Oct 29 22:05:22.604947 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.604916    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"
Oct 29 22:05:22.668565 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:05:22.668539    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f87fff_d6eb_4aa9_b6b5_cac68c2fed28.slice/crio-2fa7699370faab844340a9086acc265d08ad8582b624fe0096aa28ad5dfc4a57 WatchSource:0}: Error finding container 2fa7699370faab844340a9086acc265d08ad8582b624fe0096aa28ad5dfc4a57: Status 404 returned error can't find the container with id 2fa7699370faab844340a9086acc265d08ad8582b624fe0096aa28ad5dfc4a57
Oct 29 22:05:22.668914 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.668889    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"]
Oct 29 22:05:22.678516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.678490    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-kube-api-access-k7926\" (UniqueName: \"kubernetes.io/projected/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-kube-api-access-k7926\") pod \"26bfc7bc-ef5e-4df2-998b-bdc774f39f89\" (UID: \"26bfc7bc-ef5e-4df2-998b-bdc774f39f89\") "
Oct 29 22:05:22.678695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.678682    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-plugins-config-volume\") pod \"26bfc7bc-ef5e-4df2-998b-bdc774f39f89\" (UID: \"26bfc7bc-ef5e-4df2-998b-bdc774f39f89\") "
Oct 29 22:05:22.679069 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.679041    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-plugins-config-volume" pod "26bfc7bc-ef5e-4df2-998b-bdc774f39f89" (UID: "26bfc7bc-ef5e-4df2-998b-bdc774f39f89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:05:22.679353 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.679284    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:22.680040 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.680020    7082 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:05:22.680823 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.680800    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-kube-api-access-k7926" pod "26bfc7bc-ef5e-4df2-998b-bdc774f39f89" (UID: "26bfc7bc-ef5e-4df2-998b-bdc774f39f89"). InnerVolumeSpecName "kube-api-access-k7926". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:05:22.696313 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.696288    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"
Oct 29 22:05:22.780910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:22.780865    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7926\" (UniqueName: \"kubernetes.io/projected/26bfc7bc-ef5e-4df2-998b-bdc774f39f89-kube-api-access-k7926\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:05:23.034208 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:05:23.034176    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc1d583_b4dc_4d4b_99c4_c08a945cdad9.slice/crio-2f58e75e32e0be0be5491e427951be7691bfd302a23aa82ab854dab57827dc7d WatchSource:0}: Error finding container 2f58e75e32e0be0be5491e427951be7691bfd302a23aa82ab854dab57827dc7d: Status 404 returned error can't find the container with id 2f58e75e32e0be0be5491e427951be7691bfd302a23aa82ab854dab57827dc7d
Oct 29 22:05:23.035221 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:23.035174    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"]
Oct 29 22:05:23.109940 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:23.109894    7082 generic.go:356] "Generic (PLEG): container finished" podID="26bfc7bc-ef5e-4df2-998b-bdc774f39f89" containerID="62e3f8b61fe46b3b51fd2cb1a2bc7e91ab091fce77b94ea5bf9ea7cff9abb4f5" exitCode=0
Oct 29 22:05:23.110100 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:23.110077    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz" event={"ID":"26bfc7bc-ef5e-4df2-998b-bdc774f39f89","Type":"ContainerDied","Data":"62e3f8b61fe46b3b51fd2cb1a2bc7e91ab091fce77b94ea5bf9ea7cff9abb4f5"}
Oct 29 22:05:23.110160 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:23.110143    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz" event={"ID":"26bfc7bc-ef5e-4df2-998b-bdc774f39f89","Type":"ContainerDied","Data":"f281df2ed6fcf424dd00e007468eb2db8b91e17681d1f02ecd2871389f27c09f"}
Oct 29 22:05:23.110206 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:23.110177    7082 scope.go:122] "RemoveContainer" containerID="62e3f8b61fe46b3b51fd2cb1a2bc7e91ab091fce77b94ea5bf9ea7cff9abb4f5"
Oct 29 22:05:23.110682 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:23.110662    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"
Oct 29 22:05:23.117460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:23.117434    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4" event={"ID":"17f87fff-d6eb-4aa9-b6b5-cac68c2fed28","Type":"ContainerStarted","Data":"2fa7699370faab844340a9086acc265d08ad8582b624fe0096aa28ad5dfc4a57"}
Oct 29 22:05:23.120889 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:23.120826    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4" event={"ID":"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9","Type":"ContainerStarted","Data":"2f58e75e32e0be0be5491e427951be7691bfd302a23aa82ab854dab57827dc7d"}
Oct 29 22:05:23.146546 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:23.146524    7082 scope.go:122] "RemoveContainer" containerID="62e3f8b61fe46b3b51fd2cb1a2bc7e91ab091fce77b94ea5bf9ea7cff9abb4f5"
Oct 29 22:05:23.146932 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:05:23.146901    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e3f8b61fe46b3b51fd2cb1a2bc7e91ab091fce77b94ea5bf9ea7cff9abb4f5\": container with ID starting with 62e3f8b61fe46b3b51fd2cb1a2bc7e91ab091fce77b94ea5bf9ea7cff9abb4f5 not found: ID does not exist" containerID="62e3f8b61fe46b3b51fd2cb1a2bc7e91ab091fce77b94ea5bf9ea7cff9abb4f5"
Oct 29 22:05:23.147014 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:23.146945    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e3f8b61fe46b3b51fd2cb1a2bc7e91ab091fce77b94ea5bf9ea7cff9abb4f5"} err="failed to get container status \"62e3f8b61fe46b3b51fd2cb1a2bc7e91ab091fce77b94ea5bf9ea7cff9abb4f5\": rpc error: code = NotFound desc = could not find container \"62e3f8b61fe46b3b51fd2cb1a2bc7e91ab091fce77b94ea5bf9ea7cff9abb4f5\": container with ID starting with 62e3f8b61fe46b3b51fd2cb1a2bc7e91ab091fce77b94ea5bf9ea7cff9abb4f5 not found: ID does not exist"
Oct 29 22:05:23.193080 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:23.193055    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"]
Oct 29 22:05:23.214160 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:23.214134    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6df539-748f99ddcd-cckfz"]
Oct 29 22:05:23.643759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:23.643718    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26bfc7bc-ef5e-4df2-998b-bdc774f39f89" path="/var/lib/kubelet/pods/26bfc7bc-ef5e-4df2-998b-bdc774f39f89/volumes"
Oct 29 22:05:25.152080 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:25.152035    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4" event={"ID":"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9","Type":"ContainerStarted","Data":"50729da47df72247a6059b135606b119541341d6b11adfee83496c21f732b5ec"}
Oct 29 22:05:25.152457 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:25.152146    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"
Oct 29 22:05:25.154392 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:25.154332    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4" event={"ID":"17f87fff-d6eb-4aa9-b6b5-cac68c2fed28","Type":"ContainerStarted","Data":"c41051adac08c588f4cecb0bc345afa178be152f9e6fd9a812e7aa3f48d771f0"}
Oct 29 22:05:25.154534 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:25.154442    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"
Oct 29 22:05:25.208606 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:25.208551    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4" podStartSLOduration=2.251750113 podStartE2EDuration="4.208532203s" totalImagesPullingTime="1.95678209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:25.195977682 +0000 UTC m=+1097.696019982" watchObservedRunningTime="2026-10-29 22:05:25.208532203 +0000 UTC m=+1097.708574503"
Oct 29 22:05:25.242838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:25.242790    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4" podStartSLOduration=2.173519422 podStartE2EDuration="4.242778077s" totalImagesPullingTime="2.069258655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:25.237822765 +0000 UTC m=+1097.737865065" watchObservedRunningTime="2026-10-29 22:05:25.242778077 +0000 UTC m=+1097.742820374"
Oct 29 22:05:28.418940 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:05:28.418900    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:05:28.678268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:28.678191    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"
Oct 29 22:05:28.678268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:28.678266    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"
Oct 29 22:05:33.162881 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.162848    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"]
Oct 29 22:05:33.164323 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.164298    7082 cpu_manager.go:414] "containerMap: removing container" podUID="26bfc7bc-ef5e-4df2-998b-bdc774f39f89" containerName="payload-processing"
Oct 29 22:05:33.164323 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.164321    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26bfc7bc-ef5e-4df2-998b-bdc774f39f89" containerName="payload-processing"
Oct 29 22:05:33.164485 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.164444    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="26bfc7bc-ef5e-4df2-998b-bdc774f39f89" containerName="payload-processing"
Oct 29 22:05:33.168262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.168238    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.194495 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.194453    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f37e68-0688-4368-9323-a03970491ce9-istio-podinfo\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.195779 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.195347    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-credential-socket\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.195779 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.195720    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-envoy\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.195931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.195832    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-istio-token\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.195931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.195857    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-socket\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.195931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.195909    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f37e68-0688-4368-9323-a03970491ce9-istiod-ca-cert\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.196154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.196007    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqlbm\" (UniqueName: \"kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-kube-api-access-xqlbm\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.196154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.196034    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-certs\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.196154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.196124    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-data\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.207934 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.207905    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"]
Oct 29 22:05:33.247300 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.247264    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a26f29-openshift-default-dockercfg-x5md6\""
Oct 29 22:05:33.356731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.326730    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-credential-socket\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.356731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.326779    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-envoy\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.356731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.326889    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-istio-token\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.356731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.326913    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-socket\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.356731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.326962    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f37e68-0688-4368-9323-a03970491ce9-istiod-ca-cert\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.356731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.327068    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqlbm\" (UniqueName: \"kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-kube-api-access-xqlbm\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.356731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.327092    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-certs\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.356731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.327174    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-data\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.356731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.327295    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f37e68-0688-4368-9323-a03970491ce9-istio-podinfo\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.356731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.332902    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-socket\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.356731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.345263    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-credential-socket\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.437698 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.432573    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-certs\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.437698 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.433277    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f37e68-0688-4368-9323-a03970491ce9-istiod-ca-cert\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.494417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.442094    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-istio-token\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.494417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.444205    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-data\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.591716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.591526    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f37e68-0688-4368-9323-a03970491ce9-istio-podinfo\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.596097 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.596064    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-envoy\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.631389 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.619592    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqlbm\" (UniqueName: \"kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-kube-api-access-xqlbm\") pod \"e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") " pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:33.794708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:33.790020    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:34.305252 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:05:34.305217    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f37e68_0688_4368_9323_a03970491ce9.slice/crio-47de5575b1381c9bc2e434b4f237e84ed64380af4290bbb764eb344accad1fde WatchSource:0}: Error finding container 47de5575b1381c9bc2e434b4f237e84ed64380af4290bbb764eb344accad1fde: Status 404 returned error can't find the container with id 47de5575b1381c9bc2e434b4f237e84ed64380af4290bbb764eb344accad1fde
Oct 29 22:05:34.308286 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:34.308260    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"]
Oct 29 22:05:34.308588 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:34.308560    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:05:34.308697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:34.308644    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:05:34.308697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:34.308676    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:05:34.583699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:34.582693    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj" event={"ID":"35f37e68-0688-4368-9323-a03970491ce9","Type":"ContainerStarted","Data":"21d6acfe14098794b9dd99282c34622851f0d9cc1332e96c6a5ea81a65d66de9"}
Oct 29 22:05:34.583699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:34.582761    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj" event={"ID":"35f37e68-0688-4368-9323-a03970491ce9","Type":"ContainerStarted","Data":"47de5575b1381c9bc2e434b4f237e84ed64380af4290bbb764eb344accad1fde"}
Oct 29 22:05:34.640885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:34.640828    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj" podStartSLOduration=1.640809828 podStartE2EDuration="1.640809828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:34.63107257 +0000 UTC m=+1107.131114893" watchObservedRunningTime="2026-10-29 22:05:34.640809828 +0000 UTC m=+1107.140852133"
Oct 29 22:05:34.794679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:34.793727    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:35.040841 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:35.040758    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"]
Oct 29 22:05:35.045915 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:35.045892    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0563756-7482d"]
Oct 29 22:05:35.784920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:35.784884    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f42909-c84a-49fa-97aa-d65fb90ff51b" path="/var/lib/kubelet/pods/02f42909-c84a-49fa-97aa-d65fb90ff51b/volumes"
Oct 29 22:05:35.800729 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:35.800689    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:36.165837 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:36.165808    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"
Oct 29 22:05:36.166022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:36.165972    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"
Oct 29 22:05:36.694055 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:36.694024    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:36.695037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:36.695012    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:05:38.061186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:38.061151    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"]
Oct 29 22:05:38.072474 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:38.072448    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45de20f3-8ljw9"]
Oct 29 22:05:39.996664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:39.992125    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ded71e-f4fa-4e66-a501-6f8792598ee2" path="/var/lib/kubelet/pods/43ded71e-f4fa-4e66-a501-6f8792598ee2/volumes"
Oct 29 22:05:40.386804 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.386772    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"]
Oct 29 22:05:40.566652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.534476    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-certs\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.566652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.534684    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9szb2\" (UniqueName: \"kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-kube-api-access-9szb2\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.566652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.534723    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef300af-2bfa-4891-9c03-e60646549fc3-istiod-ca-cert\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.566652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.534764    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-podinfo\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.566652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.534804    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-credential-socket\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.566652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.534835    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-data\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.566652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.534938    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-envoy\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.566652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.534969    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-socket\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.566652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.535064    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-token\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.566652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.550885    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.599650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.579697    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"]
Oct 29 22:05:40.599650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.582547    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-301fcc-openshift-default-dockercfg-nn7lv\""
Oct 29 22:05:40.636730 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.636698    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-podinfo\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.637031 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.636978    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-credential-socket\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.637184 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.637165    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-data\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.637582 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.637456    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-credential-socket\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.637699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.637504    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-data\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.637822 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.637803    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-envoy\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.637910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.637838    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-socket\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.637970 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.637916    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-token\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.638139 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.638108    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-certs\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.638251 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.638237    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9szb2\" (UniqueName: \"kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-kube-api-access-9szb2\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.638385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.638267    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef300af-2bfa-4891-9c03-e60646549fc3-istiod-ca-cert\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.638886 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.638866    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef300af-2bfa-4891-9c03-e60646549fc3-istiod-ca-cert\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.639385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.639358    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-certs\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.639758 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.639736    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-socket\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.650497 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.645176    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:05:40.650497 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.649930    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-envoy\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.654854 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.652116    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-podinfo\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.669391 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.667460    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-token\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:40.685415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:40.685351    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9szb2\" (UniqueName: \"kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-kube-api-access-9szb2\") pod \"e2e-api-b-301fcc-openshift-default-767d445955-5mlgj\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") " pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:41.018704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:41.017896    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:41.787454 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:41.787420    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"]
Oct 29 22:05:41.898691 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:41.897204    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:05:41.898691 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:41.897312    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:05:41.898691 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:41.897354    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:05:42.049212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:42.049173    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" event={"ID":"9ef300af-2bfa-4891-9c03-e60646549fc3","Type":"ContainerStarted","Data":"1fd44327cbb545c4d41ade09713443d5c4753e9a506da6aeaee54fae7098ed47"}
Oct 29 22:05:43.064896 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:43.064853    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" event={"ID":"9ef300af-2bfa-4891-9c03-e60646549fc3","Type":"ContainerStarted","Data":"3f3cf7f4c8f056359379673e4821b910c35dc16e92827754b5193f25c47f731a"}
Oct 29 22:05:43.137560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:43.137497    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" podStartSLOduration=3.137479537 podStartE2EDuration="3.137479537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:43.129939016 +0000 UTC m=+1115.629981316" watchObservedRunningTime="2026-10-29 22:05:43.137479537 +0000 UTC m=+1115.637521835"
Oct 29 22:05:44.019230 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:44.019200    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:44.026772 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:44.026740    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:44.074913 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:44.074884    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:44.076715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:44.076619    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:05:49.149163 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:49.149118    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"]
Oct 29 22:05:49.154616 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:49.154596    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"
Oct 29 22:05:49.174678 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:49.174651    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"]
Oct 29 22:05:49.178662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:49.178607    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-301fcc\""
Oct 29 22:05:49.179617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:49.179598    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-301fcc-dockercfg-txkh5\""
Oct 29 22:05:49.332426 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:49.326904    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07e2d2d-0845-4243-82ac-6a17427f2290-plugins-config-volume\") pod \"payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk\" (UID: \"e07e2d2d-0845-4243-82ac-6a17427f2290\") " pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"
Oct 29 22:05:49.332426 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:49.327211    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cjkt\" (UniqueName: \"kubernetes.io/projected/e07e2d2d-0845-4243-82ac-6a17427f2290-kube-api-access-9cjkt\") pod \"payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk\" (UID: \"e07e2d2d-0845-4243-82ac-6a17427f2290\") " pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"
Oct 29 22:05:49.447704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:49.441011    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjkt\" (UniqueName: \"kubernetes.io/projected/e07e2d2d-0845-4243-82ac-6a17427f2290-kube-api-access-9cjkt\") pod \"payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk\" (UID: \"e07e2d2d-0845-4243-82ac-6a17427f2290\") " pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"
Oct 29 22:05:49.447704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:49.441231    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07e2d2d-0845-4243-82ac-6a17427f2290-plugins-config-volume\") pod \"payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk\" (UID: \"e07e2d2d-0845-4243-82ac-6a17427f2290\") " pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"
Oct 29 22:05:49.447704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:49.442835    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07e2d2d-0845-4243-82ac-6a17427f2290-plugins-config-volume\") pod \"payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk\" (UID: \"e07e2d2d-0845-4243-82ac-6a17427f2290\") " pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"
Oct 29 22:05:49.752729 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:49.742013    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjkt\" (UniqueName: \"kubernetes.io/projected/e07e2d2d-0845-4243-82ac-6a17427f2290-kube-api-access-9cjkt\") pod \"payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk\" (UID: \"e07e2d2d-0845-4243-82ac-6a17427f2290\") " pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"
Oct 29 22:05:49.781391 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:49.781356    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"
Oct 29 22:05:50.104810 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:05:50.104778    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07e2d2d_0845_4243_82ac_6a17427f2290.slice/crio-8d1872e675db5c2d5bb84995fe4730f66a01fcff34bffdec60c7c7030596d980 WatchSource:0}: Error finding container 8d1872e675db5c2d5bb84995fe4730f66a01fcff34bffdec60c7c7030596d980: Status 404 returned error can't find the container with id 8d1872e675db5c2d5bb84995fe4730f66a01fcff34bffdec60c7c7030596d980
Oct 29 22:05:50.105588 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:50.105563    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"]
Oct 29 22:05:50.603684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:50.603604    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk" event={"ID":"e07e2d2d-0845-4243-82ac-6a17427f2290","Type":"ContainerStarted","Data":"443cc8e317eca3d08006e70287bb0474fd0c0b31a4b845849a34d4e8b33d0843"}
Oct 29 22:05:50.604081 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:50.603745    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk" event={"ID":"e07e2d2d-0845-4243-82ac-6a17427f2290","Type":"ContainerStarted","Data":"8d1872e675db5c2d5bb84995fe4730f66a01fcff34bffdec60c7c7030596d980"}
Oct 29 22:05:50.604081 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:50.603848    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"
Oct 29 22:05:50.645355 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:50.645299    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk" podStartSLOduration=1.6452865509999999 podStartE2EDuration="1.645286551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:50.644495338 +0000 UTC m=+1123.144537637" watchObservedRunningTime="2026-10-29 22:05:50.645286551 +0000 UTC m=+1123.145328849"
Oct 29 22:05:58.019377 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:58.019334    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"]
Oct 29 22:05:58.019377 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:58.019385    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6df539-9p8bm"]
Oct 29 22:05:58.415459 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:05:58.415417    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:05:59.039909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:59.039877    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"]
Oct 29 22:05:59.050925 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:59.050897    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15541251-hh2s2"]
Oct 29 22:05:59.703652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:59.689438    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15da3721-b9f1-4140-b963-6d08b418814c" path="/var/lib/kubelet/pods/15da3721-b9f1-4140-b963-6d08b418814c/volumes"
Oct 29 22:05:59.703652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:05:59.690235    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="163e7a73-075c-4531-9d07-21fd0de43a97" path="/var/lib/kubelet/pods/163e7a73-075c-4531-9d07-21fd0de43a97/volumes"
Oct 29 22:06:01.105357 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:01.105320    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"]
Oct 29 22:06:01.119893 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:01.119861    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aee2d99-55drw"]
Oct 29 22:06:01.653513 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:01.653478    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32abe6d7-0e48-4afd-9a10-be3881236679" path="/var/lib/kubelet/pods/32abe6d7-0e48-4afd-9a10-be3881236679/volumes"
Oct 29 22:06:01.657262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:01.657239    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"
Oct 29 22:06:08.334081 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.334048    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"]
Oct 29 22:06:08.341944 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.341919    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.363144 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.363118    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f363a3-openshift-default-dockercfg-6v2wv\""
Oct 29 22:06:08.372141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.372115    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"]
Oct 29 22:06:08.418422 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.418384    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.418725 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.418704    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.418874 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.418857    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.419075 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.419051    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.419162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.419149    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.419219 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.419189    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.419272 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.419229    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.420692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.420666    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.420936 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.420916    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzmzt\" (UniqueName: \"kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-kube-api-access-mzmzt\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.540653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.530971    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.540653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.531049    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.540653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.531087    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.540653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.531167    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.540653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.531259    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.540653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.531284    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.540653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.531332    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.540653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.531511    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.540653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.531579    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmzt\" (UniqueName: \"kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-kube-api-access-mzmzt\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.572646 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.543150    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.572646 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.543487    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.572646 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.543752    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.593705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.585231    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.594789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.594759    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.595553 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.595532    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.647885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.647825    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.656073 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.656033    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.674794 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.674762    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzmzt\" (UniqueName: \"kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-kube-api-access-mzmzt\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:08.887767 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.887677    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"]
Oct 29 22:06:08.902876 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.901989    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"
Oct 29 22:06:08.917993 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.917963    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:06:08.918992 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.918971    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:06:08.919205 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.919192    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:06:08.925172 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.925143    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:06:09.004693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:08.977683    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"]
Oct 29 22:06:09.135797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.025798    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-751a7416-2nw4w\" (UID: \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"
Oct 29 22:06:09.145378 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.136257    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2lnm\" (UniqueName: \"kubernetes.io/projected/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-kube-api-access-d2lnm\") pod \"maas-api-revoke-keys-e2e-mt-751a7416-2nw4w\" (UID: \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"
Oct 29 22:06:09.145681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.145653    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-751a7416-2nw4w\" (UID: \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"
Oct 29 22:06:09.146010 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.077296    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:09.247705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.247675    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-751a7416-2nw4w\" (UID: \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"
Oct 29 22:06:09.247835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.247808    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2lnm\" (UniqueName: \"kubernetes.io/projected/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-kube-api-access-d2lnm\") pod \"maas-api-revoke-keys-e2e-mt-751a7416-2nw4w\" (UID: \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"
Oct 29 22:06:09.247880 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.247840    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-751a7416-2nw4w\" (UID: \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"
Oct 29 22:06:09.248519 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.248498    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-751a7416-2nw4w\" (UID: \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"
Oct 29 22:06:09.251019 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.250992    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-751a7416-2nw4w\" (UID: \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"
Oct 29 22:06:09.271676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.271277    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2lnm\" (UniqueName: \"kubernetes.io/projected/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-kube-api-access-d2lnm\") pod \"maas-api-revoke-keys-e2e-mt-751a7416-2nw4w\" (UID: \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"
Oct 29 22:06:09.434685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.434559    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"
Oct 29 22:06:09.527053 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.526952    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"]
Oct 29 22:06:09.530232 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:06:09.530200    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7fa2cd_1234_4b80_b869_3ab7019d1cea.slice/crio-02848b0ca1ef1afd56f491bb74b4504283a99d22e0e668f1af8d6df11c3b73d1 WatchSource:0}: Error finding container 02848b0ca1ef1afd56f491bb74b4504283a99d22e0e668f1af8d6df11c3b73d1: Status 404 returned error can't find the container with id 02848b0ca1ef1afd56f491bb74b4504283a99d22e0e668f1af8d6df11c3b73d1
Oct 29 22:06:09.533089 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.533053    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:06:09.533190 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.533121    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:06:09.533190 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.533153    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:06:09.784684 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:06:09.782367    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9097a3_0404_42f8_93c4_1c1f9c7ba79c.slice/crio-63b085f2619ba4b4b0cdee79f4fe48135f7d7e67e102d63d8785b7bc26f07fe7 WatchSource:0}: Error finding container 63b085f2619ba4b4b0cdee79f4fe48135f7d7e67e102d63d8785b7bc26f07fe7: Status 404 returned error can't find the container with id 63b085f2619ba4b4b0cdee79f4fe48135f7d7e67e102d63d8785b7bc26f07fe7
Oct 29 22:06:09.784684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:09.782615    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"]
Oct 29 22:06:10.146915 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:10.146881    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"]
Oct 29 22:06:10.172180 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:10.172140    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"]
Oct 29 22:06:10.206079 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:10.206044    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w" event={"ID":"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c","Type":"ContainerStarted","Data":"63b085f2619ba4b4b0cdee79f4fe48135f7d7e67e102d63d8785b7bc26f07fe7"}
Oct 29 22:06:10.209232 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:10.209117    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" event={"ID":"2a7fa2cd-1234-4b80-b869-3ab7019d1cea","Type":"ContainerStarted","Data":"e3229d11c4b9a6fb6a918a8920bf4458b74679563245fb37b49261b3164cde8f"}
Oct 29 22:06:10.209232 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:10.209181    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" event={"ID":"2a7fa2cd-1234-4b80-b869-3ab7019d1cea","Type":"ContainerStarted","Data":"02848b0ca1ef1afd56f491bb74b4504283a99d22e0e668f1af8d6df11c3b73d1"}
Oct 29 22:06:10.309566 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:10.309495    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podStartSLOduration=2.309472898 podStartE2EDuration="2.309472898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:10.306174532 +0000 UTC m=+1142.806216843" watchObservedRunningTime="2026-10-29 22:06:10.309472898 +0000 UTC m=+1142.809515197"
Oct 29 22:06:11.099732 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:11.097005    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:11.156977 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:11.156942    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:11.507332 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:11.504688    7082 generic.go:356] "Generic (PLEG): container finished" podID="cf9097a3-0404-42f8-93c4-1c1f9c7ba79c" containerID="4b70a76d3517e661c134b03cc4ced542cd673f1c2d851d70573bc9ca511d3971" exitCode=0
Oct 29 22:06:11.507332 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:11.505663    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w" event={"ID":"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c","Type":"ContainerDied","Data":"4b70a76d3517e661c134b03cc4ced542cd673f1c2d851d70573bc9ca511d3971"}
Oct 29 22:06:11.507332 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:11.505946    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:11.507332 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:11.507196    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:06:12.048541 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:12.048509    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"]
Oct 29 22:06:12.057024 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:12.056995    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a077d7-vbrt9"]
Oct 29 22:06:12.708471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:12.708440    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"
Oct 29 22:06:12.774468 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:12.774386    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-openshift-ca\") pod \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\" (UID: \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\") "
Oct 29 22:06:12.774718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:12.774689    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-xks-ca\") pod \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\" (UID: \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\") "
Oct 29 22:06:12.774856 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:12.774759    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-openshift-ca" pod "cf9097a3-0404-42f8-93c4-1c1f9c7ba79c" (UID: "cf9097a3-0404-42f8-93c4-1c1f9c7ba79c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:06:12.774921 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:12.774903    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-kube-api-access-d2lnm\" (UniqueName: \"kubernetes.io/projected/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-kube-api-access-d2lnm\") pod \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\" (UID: \"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c\") "
Oct 29 22:06:12.776183 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:12.776155    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:12.777050 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:12.777022    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-xks-ca" pod "cf9097a3-0404-42f8-93c4-1c1f9c7ba79c" (UID: "cf9097a3-0404-42f8-93c4-1c1f9c7ba79c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:06:12.779690 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:12.779618    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-kube-api-access-d2lnm" pod "cf9097a3-0404-42f8-93c4-1c1f9c7ba79c" (UID: "cf9097a3-0404-42f8-93c4-1c1f9c7ba79c"). InnerVolumeSpecName "kube-api-access-d2lnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:06:12.878466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:12.878433    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:12.878466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:12.878468    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2lnm\" (UniqueName: \"kubernetes.io/projected/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c-kube-api-access-d2lnm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:13.530028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:13.529990    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"
Oct 29 22:06:13.530028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:13.530010    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w" event={"ID":"cf9097a3-0404-42f8-93c4-1c1f9c7ba79c","Type":"ContainerDied","Data":"63b085f2619ba4b4b0cdee79f4fe48135f7d7e67e102d63d8785b7bc26f07fe7"}
Oct 29 22:06:13.530311 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:13.530086    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63b085f2619ba4b4b0cdee79f4fe48135f7d7e67e102d63d8785b7bc26f07fe7"
Oct 29 22:06:13.635058 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:13.635022    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d7106f-f979-43d1-b91e-4ed51f5d2bf8" path="/var/lib/kubelet/pods/c6d7106f-f979-43d1-b91e-4ed51f5d2bf8/volumes"
Oct 29 22:06:15.213036 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:15.210696    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z" podUID="462635bd-9dd1-4132-b1a4-5eba418881f1" containerName="payload-pre-processing" containerID="cri-o://6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9" gracePeriod=25
Oct 29 22:06:15.232408 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:15.227970    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9" podUID="7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d" containerName="payload-processing" containerID="cri-o://b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0" gracePeriod=25
Oct 29 22:06:16.027553 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:06:16.026689    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462635bd_9dd1_4132_b1a4_5eba418881f1.slice/crio-conmon-6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4fe5f5_c566_47cd_b3ba_abc8c8a2c96d.slice/crio-conmon-b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462635bd_9dd1_4132_b1a4_5eba418881f1.slice/crio-6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4fe5f5_c566_47cd_b3ba_abc8c8a2c96d.slice/crio-b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:06:16.027553 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:06:16.026759    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4fe5f5_c566_47cd_b3ba_abc8c8a2c96d.slice/crio-b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462635bd_9dd1_4132_b1a4_5eba418881f1.slice/crio-6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462635bd_9dd1_4132_b1a4_5eba418881f1.slice/crio-conmon-6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:06:16.027553 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:06:16.026836    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4fe5f5_c566_47cd_b3ba_abc8c8a2c96d.slice/crio-conmon-b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4fe5f5_c566_47cd_b3ba_abc8c8a2c96d.slice/crio-b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462635bd_9dd1_4132_b1a4_5eba418881f1.slice/crio-conmon-6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462635bd_9dd1_4132_b1a4_5eba418881f1.slice/crio-6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:06:16.068030 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.067995    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"]
Oct 29 22:06:16.079899 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.079500    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cffbe85-hx5zr"]
Oct 29 22:06:16.350237 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.350212    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"
Oct 29 22:06:16.407811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.407782    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462635bd-9dd1-4132-b1a4-5eba418881f1-kube-api-access-q58wm\" (UniqueName: \"kubernetes.io/projected/462635bd-9dd1-4132-b1a4-5eba418881f1-kube-api-access-q58wm\") pod \"462635bd-9dd1-4132-b1a4-5eba418881f1\" (UID: \"462635bd-9dd1-4132-b1a4-5eba418881f1\") "
Oct 29 22:06:16.408035 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.408016    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/462635bd-9dd1-4132-b1a4-5eba418881f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462635bd-9dd1-4132-b1a4-5eba418881f1-plugins-config-volume\") pod \"462635bd-9dd1-4132-b1a4-5eba418881f1\" (UID: \"462635bd-9dd1-4132-b1a4-5eba418881f1\") "
Oct 29 22:06:16.408681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.408435    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462635bd-9dd1-4132-b1a4-5eba418881f1-plugins-config-volume" pod "462635bd-9dd1-4132-b1a4-5eba418881f1" (UID: "462635bd-9dd1-4132-b1a4-5eba418881f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:06:16.408909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.408888    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462635bd-9dd1-4132-b1a4-5eba418881f1-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:16.410513 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.410489    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462635bd-9dd1-4132-b1a4-5eba418881f1-kube-api-access-q58wm" pod "462635bd-9dd1-4132-b1a4-5eba418881f1" (UID: "462635bd-9dd1-4132-b1a4-5eba418881f1"). InnerVolumeSpecName "kube-api-access-q58wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:06:16.438942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.438922    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"
Oct 29 22:06:16.557214 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.557179    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-kube-api-access-lzvw9\" (UniqueName: \"kubernetes.io/projected/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-kube-api-access-lzvw9\") pod \"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d\" (UID: \"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d\") "
Oct 29 22:06:16.570680 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.558053    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-plugins-config-volume\") pod \"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d\" (UID: \"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d\") "
Oct 29 22:06:16.579689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.571871    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q58wm\" (UniqueName: \"kubernetes.io/projected/462635bd-9dd1-4132-b1a4-5eba418881f1-kube-api-access-q58wm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:16.579898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.565097    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-plugins-config-volume" pod "7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d" (UID: "7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:06:16.780800 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.737527    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:16.780800 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.742560    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-kube-api-access-lzvw9" pod "7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d" (UID: "7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d"). InnerVolumeSpecName "kube-api-access-lzvw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:06:16.805603 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.803726    7082 generic.go:356] "Generic (PLEG): container finished" podID="7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d" containerID="b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0" exitCode=0
Oct 29 22:06:16.805603 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.803898    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"
Oct 29 22:06:16.811652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.807804    7082 generic.go:356] "Generic (PLEG): container finished" podID="462635bd-9dd1-4132-b1a4-5eba418881f1" containerID="6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9" exitCode=0
Oct 29 22:06:16.811652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.807968    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"
Oct 29 22:06:16.818561 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.814519    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9" event={"ID":"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d","Type":"ContainerDied","Data":"b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0"}
Oct 29 22:06:16.820447 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.814586    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9" event={"ID":"7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d","Type":"ContainerDied","Data":"e3dbddd24593257226bec4da9e87f6d7b0b2802ecc705ccca8331c44cd58548b"}
Oct 29 22:06:16.820447 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.819096    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z" event={"ID":"462635bd-9dd1-4132-b1a4-5eba418881f1","Type":"ContainerDied","Data":"6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9"}
Oct 29 22:06:16.820447 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.819127    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z" event={"ID":"462635bd-9dd1-4132-b1a4-5eba418881f1","Type":"ContainerDied","Data":"afa87a6c01f47cb0260a40f1a9d83e13dd7cf5557fc30529b22790615692f482"}
Oct 29 22:06:16.820447 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.819157    7082 scope.go:122] "RemoveContainer" containerID="b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0"
Oct 29 22:06:16.843706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.841913    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzvw9\" (UniqueName: \"kubernetes.io/projected/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d-kube-api-access-lzvw9\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:16.845870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.845846    7082 scope.go:122] "RemoveContainer" containerID="b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0"
Oct 29 22:06:16.846243 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:06:16.846206    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0\": container with ID starting with b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0 not found: ID does not exist" containerID="b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0"
Oct 29 22:06:16.846450 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.846415    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0"} err="failed to get container status \"b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0\": rpc error: code = NotFound desc = could not find container \"b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0\": container with ID starting with b140774b69ecc0406e659a64036125fe51a8b5ccad437c9bcda9ce25a88075d0 not found: ID does not exist"
Oct 29 22:06:16.846579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.846564    7082 scope.go:122] "RemoveContainer" containerID="6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9"
Oct 29 22:06:16.862047 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.862020    7082 scope.go:122] "RemoveContainer" containerID="6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9"
Oct 29 22:06:16.862356 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:06:16.862335    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9\": container with ID starting with 6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9 not found: ID does not exist" containerID="6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9"
Oct 29 22:06:16.862429 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.862365    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9"} err="failed to get container status \"6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9\": rpc error: code = NotFound desc = could not find container \"6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9\": container with ID starting with 6367bac69b2174fee377b922ac1cf56fb68b4d2c19fefa77a9900b7bfa9731e9 not found: ID does not exist"
Oct 29 22:06:16.927293 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.927259    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"]
Oct 29 22:06:16.932293 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.932270    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-b866b75b8-gl4m9"]
Oct 29 22:06:16.964408 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.964376    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"]
Oct 29 22:06:16.968790 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:16.968765    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-66d786fb59-jz84z"]
Oct 29 22:06:17.705654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:17.702796    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462635bd-9dd1-4132-b1a4-5eba418881f1" path="/var/lib/kubelet/pods/462635bd-9dd1-4132-b1a4-5eba418881f1/volumes"
Oct 29 22:06:17.705654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:17.703614    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d" path="/var/lib/kubelet/pods/7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d/volumes"
Oct 29 22:06:17.705654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:17.704466    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e98faa-9f44-41f2-af30-47a0260f1ab1" path="/var/lib/kubelet/pods/f3e98faa-9f44-41f2-af30-47a0260f1ab1/volumes"
Oct 29 22:06:19.078314 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:19.078277    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"]
Oct 29 22:06:21.472093 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.472059    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"]
Oct 29 22:06:21.472882 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.472866    7082 cpu_manager.go:414] "containerMap: removing container" podUID="cf9097a3-0404-42f8-93c4-1c1f9c7ba79c" containerName="revoke-keys"
Oct 29 22:06:21.472937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.472882    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9097a3-0404-42f8-93c4-1c1f9c7ba79c" containerName="revoke-keys"
Oct 29 22:06:21.472937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.472901    7082 cpu_manager.go:414] "containerMap: removing container" podUID="7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d" containerName="payload-processing"
Oct 29 22:06:21.472937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.472907    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d" containerName="payload-processing"
Oct 29 22:06:21.472937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.472930    7082 cpu_manager.go:414] "containerMap: removing container" podUID="462635bd-9dd1-4132-b1a4-5eba418881f1" containerName="payload-pre-processing"
Oct 29 22:06:21.472937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.472936    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462635bd-9dd1-4132-b1a4-5eba418881f1" containerName="payload-pre-processing"
Oct 29 22:06:21.473091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.473053    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9097a3-0404-42f8-93c4-1c1f9c7ba79c" containerName="revoke-keys"
Oct 29 22:06:21.473091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.473064    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4fe5f5-c566-47cd-b3ba-abc8c8a2c96d" containerName="payload-processing"
Oct 29 22:06:21.473091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.473074    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="462635bd-9dd1-4132-b1a4-5eba418881f1" containerName="payload-pre-processing"
Oct 29 22:06:21.478345 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.478325    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"
Oct 29 22:06:21.505320 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.505291    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:06:21.505485 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.505357    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:06:21.505485 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.505438    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:06:21.505705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.505559    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:06:21.514343 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.514315    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"]
Oct 29 22:06:21.569440 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.569408    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcade7ab-9w9wk\" (UID: \"08d562be-eb07-4d4b-be16-9cadd639f62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"
Oct 29 22:06:21.569601 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.569445    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw4vl\" (UniqueName: \"kubernetes.io/projected/08d562be-eb07-4d4b-be16-9cadd639f62d-kube-api-access-hw4vl\") pod \"maas-api-revoke-sub-fcade7ab-9w9wk\" (UID: \"08d562be-eb07-4d4b-be16-9cadd639f62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"
Oct 29 22:06:21.569664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.569598    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcade7ab-9w9wk\" (UID: \"08d562be-eb07-4d4b-be16-9cadd639f62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"
Oct 29 22:06:21.670733 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.670705    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcade7ab-9w9wk\" (UID: \"08d562be-eb07-4d4b-be16-9cadd639f62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"
Oct 29 22:06:21.671322 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.671295    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcade7ab-9w9wk\" (UID: \"08d562be-eb07-4d4b-be16-9cadd639f62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"
Oct 29 22:06:21.671428 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.671337    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw4vl\" (UniqueName: \"kubernetes.io/projected/08d562be-eb07-4d4b-be16-9cadd639f62d-kube-api-access-hw4vl\") pod \"maas-api-revoke-sub-fcade7ab-9w9wk\" (UID: \"08d562be-eb07-4d4b-be16-9cadd639f62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"
Oct 29 22:06:21.671428 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.671411    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcade7ab-9w9wk\" (UID: \"08d562be-eb07-4d4b-be16-9cadd639f62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"
Oct 29 22:06:21.673760 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.673730    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcade7ab-9w9wk\" (UID: \"08d562be-eb07-4d4b-be16-9cadd639f62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"
Oct 29 22:06:21.731381 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.731303    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw4vl\" (UniqueName: \"kubernetes.io/projected/08d562be-eb07-4d4b-be16-9cadd639f62d-kube-api-access-hw4vl\") pod \"maas-api-revoke-sub-fcade7ab-9w9wk\" (UID: \"08d562be-eb07-4d4b-be16-9cadd639f62d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"
Oct 29 22:06:21.794014 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:21.793958    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"
Oct 29 22:06:22.138076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:22.138046    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"]
Oct 29 22:06:22.143690 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:06:22.143664    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d562be_eb07_4d4b_be16_9cadd639f62d.slice/crio-589955410e575dc6e9ebcc37ecd0aebf9bf866d6f71f57195ffec9570739a881 WatchSource:0}: Error finding container 589955410e575dc6e9ebcc37ecd0aebf9bf866d6f71f57195ffec9570739a881: Status 404 returned error can't find the container with id 589955410e575dc6e9ebcc37ecd0aebf9bf866d6f71f57195ffec9570739a881
Oct 29 22:06:23.126614 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:23.126569    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk" event={"ID":"08d562be-eb07-4d4b-be16-9cadd639f62d","Type":"ContainerStarted","Data":"080f0671dc710ddbdd6e8e9b25565bf913b6495eb50ed049fc7573a5faaae708"}
Oct 29 22:06:23.127083 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:23.126686    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk" event={"ID":"08d562be-eb07-4d4b-be16-9cadd639f62d","Type":"ContainerStarted","Data":"589955410e575dc6e9ebcc37ecd0aebf9bf866d6f71f57195ffec9570739a881"}
Oct 29 22:06:23.171596 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:23.171543    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk" podStartSLOduration=2.171527228 podStartE2EDuration="2.171527228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:23.167802702 +0000 UTC m=+1155.667845001" watchObservedRunningTime="2026-10-29 22:06:23.171527228 +0000 UTC m=+1155.671569534"
Oct 29 22:06:24.079530 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:24.079480    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr" podUID="fc8da301-9bbf-4985-b81e-02e1360ab153" containerName="payload-processing" containerID="cri-o://819862e21ce4ee98f8d718fb083ade3524862f24463e28a134696d77aa7c6607" gracePeriod=25
Oct 29 22:06:24.137787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:24.137756    7082 generic.go:356] "Generic (PLEG): container finished" podID="08d562be-eb07-4d4b-be16-9cadd639f62d" containerID="080f0671dc710ddbdd6e8e9b25565bf913b6495eb50ed049fc7573a5faaae708" exitCode=0
Oct 29 22:06:24.138155 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:24.137821    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk" event={"ID":"08d562be-eb07-4d4b-be16-9cadd639f62d","Type":"ContainerDied","Data":"080f0671dc710ddbdd6e8e9b25565bf913b6495eb50ed049fc7573a5faaae708"}
Oct 29 22:06:24.818802 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:24.818776    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"
Oct 29 22:06:24.898583 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:24.898488    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8da301-9bbf-4985-b81e-02e1360ab153-kube-api-access-txd2v\" (UniqueName: \"kubernetes.io/projected/fc8da301-9bbf-4985-b81e-02e1360ab153-kube-api-access-txd2v\") pod \"fc8da301-9bbf-4985-b81e-02e1360ab153\" (UID: \"fc8da301-9bbf-4985-b81e-02e1360ab153\") "
Oct 29 22:06:24.898809 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:24.898721    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc8da301-9bbf-4985-b81e-02e1360ab153-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8da301-9bbf-4985-b81e-02e1360ab153-plugins-config-volume\") pod \"fc8da301-9bbf-4985-b81e-02e1360ab153\" (UID: \"fc8da301-9bbf-4985-b81e-02e1360ab153\") "
Oct 29 22:06:24.899150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:24.899125    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8da301-9bbf-4985-b81e-02e1360ab153-plugins-config-volume" pod "fc8da301-9bbf-4985-b81e-02e1360ab153" (UID: "fc8da301-9bbf-4985-b81e-02e1360ab153"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:06:24.899988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:24.899949    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8da301-9bbf-4985-b81e-02e1360ab153-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:24.901243 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:24.901205    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8da301-9bbf-4985-b81e-02e1360ab153-kube-api-access-txd2v" pod "fc8da301-9bbf-4985-b81e-02e1360ab153" (UID: "fc8da301-9bbf-4985-b81e-02e1360ab153"). InnerVolumeSpecName "kube-api-access-txd2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:06:25.001284 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.001254    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txd2v\" (UniqueName: \"kubernetes.io/projected/fc8da301-9bbf-4985-b81e-02e1360ab153-kube-api-access-txd2v\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:25.153703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.153583    7082 generic.go:356] "Generic (PLEG): container finished" podID="fc8da301-9bbf-4985-b81e-02e1360ab153" containerID="819862e21ce4ee98f8d718fb083ade3524862f24463e28a134696d77aa7c6607" exitCode=0
Oct 29 22:06:25.154117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.153692    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr" event={"ID":"fc8da301-9bbf-4985-b81e-02e1360ab153","Type":"ContainerDied","Data":"819862e21ce4ee98f8d718fb083ade3524862f24463e28a134696d77aa7c6607"}
Oct 29 22:06:25.154117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.153745    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"
Oct 29 22:06:25.154117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.153774    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr" event={"ID":"fc8da301-9bbf-4985-b81e-02e1360ab153","Type":"ContainerDied","Data":"b4a2a3846b298f2aca9a4172f80912c3013080d1b72b6c9c4a3978301945db0f"}
Oct 29 22:06:25.154117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.153807    7082 scope.go:122] "RemoveContainer" containerID="819862e21ce4ee98f8d718fb083ade3524862f24463e28a134696d77aa7c6607"
Oct 29 22:06:25.169019 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.168994    7082 scope.go:122] "RemoveContainer" containerID="819862e21ce4ee98f8d718fb083ade3524862f24463e28a134696d77aa7c6607"
Oct 29 22:06:25.169334 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:06:25.169306    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819862e21ce4ee98f8d718fb083ade3524862f24463e28a134696d77aa7c6607\": container with ID starting with 819862e21ce4ee98f8d718fb083ade3524862f24463e28a134696d77aa7c6607 not found: ID does not exist" containerID="819862e21ce4ee98f8d718fb083ade3524862f24463e28a134696d77aa7c6607"
Oct 29 22:06:25.169397 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.169339    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819862e21ce4ee98f8d718fb083ade3524862f24463e28a134696d77aa7c6607"} err="failed to get container status \"819862e21ce4ee98f8d718fb083ade3524862f24463e28a134696d77aa7c6607\": rpc error: code = NotFound desc = could not find container \"819862e21ce4ee98f8d718fb083ade3524862f24463e28a134696d77aa7c6607\": container with ID starting with 819862e21ce4ee98f8d718fb083ade3524862f24463e28a134696d77aa7c6607 not found: ID does not exist"
Oct 29 22:06:25.209224 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.209190    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"]
Oct 29 22:06:25.216708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.216683    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-751a7416-8f4b79588-pdlnr"]
Oct 29 22:06:25.515021 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.514931    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"
Oct 29 22:06:25.558856 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.558821    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-openshift-ca\") pod \"08d562be-eb07-4d4b-be16-9cadd639f62d\" (UID: \"08d562be-eb07-4d4b-be16-9cadd639f62d\") "
Oct 29 22:06:25.559041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.558989    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d562be-eb07-4d4b-be16-9cadd639f62d-kube-api-access-hw4vl\" (UniqueName: \"kubernetes.io/projected/08d562be-eb07-4d4b-be16-9cadd639f62d-kube-api-access-hw4vl\") pod \"08d562be-eb07-4d4b-be16-9cadd639f62d\" (UID: \"08d562be-eb07-4d4b-be16-9cadd639f62d\") "
Oct 29 22:06:25.559041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.559037    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-xks-ca\") pod \"08d562be-eb07-4d4b-be16-9cadd639f62d\" (UID: \"08d562be-eb07-4d4b-be16-9cadd639f62d\") "
Oct 29 22:06:25.559493 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.559213    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-openshift-ca" pod "08d562be-eb07-4d4b-be16-9cadd639f62d" (UID: "08d562be-eb07-4d4b-be16-9cadd639f62d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:06:25.559942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.559920    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:25.564618 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.564588    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-xks-ca" pod "08d562be-eb07-4d4b-be16-9cadd639f62d" (UID: "08d562be-eb07-4d4b-be16-9cadd639f62d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:06:25.564831 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.564791    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d562be-eb07-4d4b-be16-9cadd639f62d-kube-api-access-hw4vl" pod "08d562be-eb07-4d4b-be16-9cadd639f62d" (UID: "08d562be-eb07-4d4b-be16-9cadd639f62d"). InnerVolumeSpecName "kube-api-access-hw4vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:06:25.634561 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.634532    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc8da301-9bbf-4985-b81e-02e1360ab153" path="/var/lib/kubelet/pods/fc8da301-9bbf-4985-b81e-02e1360ab153/volumes"
Oct 29 22:06:25.661449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.661425    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw4vl\" (UniqueName: \"kubernetes.io/projected/08d562be-eb07-4d4b-be16-9cadd639f62d-kube-api-access-hw4vl\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:25.661449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:25.661452    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d562be-eb07-4d4b-be16-9cadd639f62d-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:26.322138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:26.322100    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk" event={"ID":"08d562be-eb07-4d4b-be16-9cadd639f62d","Type":"ContainerDied","Data":"589955410e575dc6e9ebcc37ecd0aebf9bf866d6f71f57195ffec9570739a881"}
Oct 29 22:06:26.324575 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:26.324548    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="589955410e575dc6e9ebcc37ecd0aebf9bf866d6f71f57195ffec9570739a881"
Oct 29 22:06:26.324851 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:26.324835    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"
Oct 29 22:06:28.416918 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:06:28.416880    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:06:36.475685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.389987    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"]
Oct 29 22:06:36.475685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.391418    7082 cpu_manager.go:414] "containerMap: removing container" podUID="08d562be-eb07-4d4b-be16-9cadd639f62d" containerName="revoke-keys"
Oct 29 22:06:36.475685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.391452    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d562be-eb07-4d4b-be16-9cadd639f62d" containerName="revoke-keys"
Oct 29 22:06:36.475685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.391488    7082 cpu_manager.go:414] "containerMap: removing container" podUID="fc8da301-9bbf-4985-b81e-02e1360ab153" containerName="payload-processing"
Oct 29 22:06:36.475685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.391498    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8da301-9bbf-4985-b81e-02e1360ab153" containerName="payload-processing"
Oct 29 22:06:36.475685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.391759    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8da301-9bbf-4985-b81e-02e1360ab153" containerName="payload-processing"
Oct 29 22:06:36.475685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.391804    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d562be-eb07-4d4b-be16-9cadd639f62d" containerName="revoke-keys"
Oct 29 22:06:36.735721 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.735530    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"]
Oct 29 22:06:36.738032 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.737998    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"
Oct 29 22:06:36.765604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.764810    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-dd38c2\""
Oct 29 22:06:36.765604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.765002    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-dd38c2-dockercfg-prcxb\""
Oct 29 22:06:36.776649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.773522    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"]
Oct 29 22:06:36.810468 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.810438    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"
Oct 29 22:06:36.826066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.826024    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"]
Oct 29 22:06:36.843762 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.843713    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08aebda0-8d4c-433a-b844-2845c9a35971-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs\" (UID: \"08aebda0-8d4c-433a-b844-2845c9a35971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"
Oct 29 22:06:36.844022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.844001    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bc7ece-7212-40cb-954d-4304de2eaf2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd\" (UID: \"76bc7ece-7212-40cb-954d-4304de2eaf2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"
Oct 29 22:06:36.844459 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.844439    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr8ks\" (UniqueName: \"kubernetes.io/projected/76bc7ece-7212-40cb-954d-4304de2eaf2c-kube-api-access-gr8ks\") pod \"payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd\" (UID: \"76bc7ece-7212-40cb-954d-4304de2eaf2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"
Oct 29 22:06:36.844575 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.844557    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv46t\" (UniqueName: \"kubernetes.io/projected/08aebda0-8d4c-433a-b844-2845c9a35971-kube-api-access-qv46t\") pod \"payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs\" (UID: \"08aebda0-8d4c-433a-b844-2845c9a35971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"
Oct 29 22:06:36.946284 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.946247    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08aebda0-8d4c-433a-b844-2845c9a35971-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs\" (UID: \"08aebda0-8d4c-433a-b844-2845c9a35971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"
Oct 29 22:06:36.946284 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.946285    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bc7ece-7212-40cb-954d-4304de2eaf2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd\" (UID: \"76bc7ece-7212-40cb-954d-4304de2eaf2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"
Oct 29 22:06:36.946478 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.946452    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr8ks\" (UniqueName: \"kubernetes.io/projected/76bc7ece-7212-40cb-954d-4304de2eaf2c-kube-api-access-gr8ks\") pod \"payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd\" (UID: \"76bc7ece-7212-40cb-954d-4304de2eaf2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"
Oct 29 22:06:36.946543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.946527    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv46t\" (UniqueName: \"kubernetes.io/projected/08aebda0-8d4c-433a-b844-2845c9a35971-kube-api-access-qv46t\") pod \"payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs\" (UID: \"08aebda0-8d4c-433a-b844-2845c9a35971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"
Oct 29 22:06:36.946976 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.946955    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08aebda0-8d4c-433a-b844-2845c9a35971-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs\" (UID: \"08aebda0-8d4c-433a-b844-2845c9a35971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"
Oct 29 22:06:36.947152 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.947136    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bc7ece-7212-40cb-954d-4304de2eaf2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd\" (UID: \"76bc7ece-7212-40cb-954d-4304de2eaf2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"
Oct 29 22:06:36.969754 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.969730    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr8ks\" (UniqueName: \"kubernetes.io/projected/76bc7ece-7212-40cb-954d-4304de2eaf2c-kube-api-access-gr8ks\") pod \"payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd\" (UID: \"76bc7ece-7212-40cb-954d-4304de2eaf2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"
Oct 29 22:06:36.969871 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:36.969796    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv46t\" (UniqueName: \"kubernetes.io/projected/08aebda0-8d4c-433a-b844-2845c9a35971-kube-api-access-qv46t\") pod \"payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs\" (UID: \"08aebda0-8d4c-433a-b844-2845c9a35971\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"
Oct 29 22:06:37.093877 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:37.093841    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"
Oct 29 22:06:37.172662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:37.164436    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"
Oct 29 22:06:37.568859 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:06:37.568825    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bc7ece_7212_40cb_954d_4304de2eaf2c.slice/crio-cc15ae79d60980d6696d102792e3d27249993fba9c16cd1b07f1423363d6e6fb WatchSource:0}: Error finding container cc15ae79d60980d6696d102792e3d27249993fba9c16cd1b07f1423363d6e6fb: Status 404 returned error can't find the container with id cc15ae79d60980d6696d102792e3d27249993fba9c16cd1b07f1423363d6e6fb
Oct 29 22:06:37.569264 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:37.569088    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"]
Oct 29 22:06:37.624991 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:06:37.624957    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08aebda0_8d4c_433a_b844_2845c9a35971.slice/crio-64973ce93596ebc88b72f200ebb4a10221111bbeec7f8dc6b7ee186fc5eb4e96 WatchSource:0}: Error finding container 64973ce93596ebc88b72f200ebb4a10221111bbeec7f8dc6b7ee186fc5eb4e96: Status 404 returned error can't find the container with id 64973ce93596ebc88b72f200ebb4a10221111bbeec7f8dc6b7ee186fc5eb4e96
Oct 29 22:06:37.642371 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:37.642347    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"]
Oct 29 22:06:37.758413 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:37.758376    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd" event={"ID":"76bc7ece-7212-40cb-954d-4304de2eaf2c","Type":"ContainerStarted","Data":"2b713f22e1020f3edbe13267229b9c51c7d186685624f793629ed40c162ad97c"}
Oct 29 22:06:37.758582 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:37.758438    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd" event={"ID":"76bc7ece-7212-40cb-954d-4304de2eaf2c","Type":"ContainerStarted","Data":"cc15ae79d60980d6696d102792e3d27249993fba9c16cd1b07f1423363d6e6fb"}
Oct 29 22:06:37.758669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:37.758582    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"
Oct 29 22:06:37.760268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:37.760239    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs" event={"ID":"08aebda0-8d4c-433a-b844-2845c9a35971","Type":"ContainerStarted","Data":"30728a04fe0467c1efa51df7b1a1dd08957356dfda4d475f8255291d7ba65266"}
Oct 29 22:06:37.760394 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:37.760289    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs" event={"ID":"08aebda0-8d4c-433a-b844-2845c9a35971","Type":"ContainerStarted","Data":"64973ce93596ebc88b72f200ebb4a10221111bbeec7f8dc6b7ee186fc5eb4e96"}
Oct 29 22:06:37.760394 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:37.760316    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"
Oct 29 22:06:37.804433 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:37.804374    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd" podStartSLOduration=1.804356629 podStartE2EDuration="1.804356629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:37.803047729 +0000 UTC m=+1170.303090039" watchObservedRunningTime="2026-10-29 22:06:37.804356629 +0000 UTC m=+1170.304398929"
Oct 29 22:06:37.852288 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:37.852226    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs" podStartSLOduration=1.852209177 podStartE2EDuration="1.852209177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:37.849982203 +0000 UTC m=+1170.350024501" watchObservedRunningTime="2026-10-29 22:06:37.852209177 +0000 UTC m=+1170.352251477"
Oct 29 22:06:39.348335 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.348303    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"]
Oct 29 22:06:39.353742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.353719    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"
Oct 29 22:06:39.368514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.368495    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:06:39.380065 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.380037    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:06:39.398679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.398652    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"]
Oct 29 22:06:39.415907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.408989    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:06:39.415907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.409057    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:06:39.458827 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.458800    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk\" (UID: \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"
Oct 29 22:06:39.458980 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.458847    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5djf\" (UniqueName: \"kubernetes.io/projected/3fceded2-c12d-4d35-8039-e3fefbc01ed8-kube-api-access-l5djf\") pod \"maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk\" (UID: \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"
Oct 29 22:06:39.458980 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.458951    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk\" (UID: \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"
Oct 29 22:06:39.560638 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.560601    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk\" (UID: \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"
Oct 29 22:06:39.560812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.560675    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5djf\" (UniqueName: \"kubernetes.io/projected/3fceded2-c12d-4d35-8039-e3fefbc01ed8-kube-api-access-l5djf\") pod \"maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk\" (UID: \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"
Oct 29 22:06:39.560812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.560725    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk\" (UID: \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"
Oct 29 22:06:39.561361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.561340    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk\" (UID: \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"
Oct 29 22:06:39.563191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.563164    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk\" (UID: \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"
Oct 29 22:06:39.584537 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.584515    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5djf\" (UniqueName: \"kubernetes.io/projected/3fceded2-c12d-4d35-8039-e3fefbc01ed8-kube-api-access-l5djf\") pod \"maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk\" (UID: \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"
Oct 29 22:06:39.670821 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:39.670749    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"
Oct 29 22:06:40.028508 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:06:40.028477    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fceded2_c12d_4d35_8039_e3fefbc01ed8.slice/crio-22624d5ba1b844e1781603f82a672848da4817e5fa8588abf2f1abf8caaad15a WatchSource:0}: Error finding container 22624d5ba1b844e1781603f82a672848da4817e5fa8588abf2f1abf8caaad15a: Status 404 returned error can't find the container with id 22624d5ba1b844e1781603f82a672848da4817e5fa8588abf2f1abf8caaad15a
Oct 29 22:06:40.029052 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:40.029027    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"]
Oct 29 22:06:40.795885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:40.795798    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk" event={"ID":"3fceded2-c12d-4d35-8039-e3fefbc01ed8","Type":"ContainerStarted","Data":"0e480829ecdefd9db1337b0295e79353b729e312eb876e770eb77475955702cb"}
Oct 29 22:06:40.795885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:40.795867    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk" event={"ID":"3fceded2-c12d-4d35-8039-e3fefbc01ed8","Type":"ContainerStarted","Data":"22624d5ba1b844e1781603f82a672848da4817e5fa8588abf2f1abf8caaad15a"}
Oct 29 22:06:40.842653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:40.840085    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk" podStartSLOduration=1.8400646699999998 podStartE2EDuration="1.84006467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:40.836695165 +0000 UTC m=+1173.336737466" watchObservedRunningTime="2026-10-29 22:06:40.84006467 +0000 UTC m=+1173.340106967"
Oct 29 22:06:41.808863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:41.808832    7082 generic.go:356] "Generic (PLEG): container finished" podID="3fceded2-c12d-4d35-8039-e3fefbc01ed8" containerID="0e480829ecdefd9db1337b0295e79353b729e312eb876e770eb77475955702cb" exitCode=0
Oct 29 22:06:41.809236 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:41.808922    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk" event={"ID":"3fceded2-c12d-4d35-8039-e3fefbc01ed8","Type":"ContainerDied","Data":"0e480829ecdefd9db1337b0295e79353b729e312eb876e770eb77475955702cb"}
Oct 29 22:06:43.027805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:43.027772    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"
Oct 29 22:06:43.061414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:43.061381    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fceded2-c12d-4d35-8039-e3fefbc01ed8-kube-api-access-l5djf\" (UniqueName: \"kubernetes.io/projected/3fceded2-c12d-4d35-8039-e3fefbc01ed8-kube-api-access-l5djf\") pod \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\" (UID: \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\") "
Oct 29 22:06:43.061554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:43.061509    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-xks-ca\") pod \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\" (UID: \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\") "
Oct 29 22:06:43.061784 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:43.061757    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-openshift-ca\") pod \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\" (UID: \"3fceded2-c12d-4d35-8039-e3fefbc01ed8\") "
Oct 29 22:06:43.062170 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:43.062110    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-openshift-ca" pod "3fceded2-c12d-4d35-8039-e3fefbc01ed8" (UID: "3fceded2-c12d-4d35-8039-e3fefbc01ed8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:06:43.062758 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:43.062735    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:43.063844 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:43.063819    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fceded2-c12d-4d35-8039-e3fefbc01ed8-kube-api-access-l5djf" pod "3fceded2-c12d-4d35-8039-e3fefbc01ed8" (UID: "3fceded2-c12d-4d35-8039-e3fefbc01ed8"). InnerVolumeSpecName "kube-api-access-l5djf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:06:43.063934 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:43.063912    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-xks-ca" pod "3fceded2-c12d-4d35-8039-e3fefbc01ed8" (UID: "3fceded2-c12d-4d35-8039-e3fefbc01ed8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:06:43.164521 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:43.164487    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5djf\" (UniqueName: \"kubernetes.io/projected/3fceded2-c12d-4d35-8039-e3fefbc01ed8-kube-api-access-l5djf\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:43.164521 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:43.164515    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fceded2-c12d-4d35-8039-e3fefbc01ed8-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:06:43.836460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:43.836422    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk" event={"ID":"3fceded2-c12d-4d35-8039-e3fefbc01ed8","Type":"ContainerDied","Data":"22624d5ba1b844e1781603f82a672848da4817e5fa8588abf2f1abf8caaad15a"}
Oct 29 22:06:43.836828 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:43.836488    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22624d5ba1b844e1781603f82a672848da4817e5fa8588abf2f1abf8caaad15a"
Oct 29 22:06:43.836828 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:43.836522    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"
Oct 29 22:06:46.984696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:46.984591    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:06:47.046830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.046797    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"]
Oct 29 22:06:47.053398 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.053375    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d4314b-h5j2f"]
Oct 29 22:06:47.602564 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.602531    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"]
Oct 29 22:06:47.603715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.603690    7082 cpu_manager.go:414] "containerMap: removing container" podUID="3fceded2-c12d-4d35-8039-e3fefbc01ed8" containerName="revoke-keys"
Oct 29 22:06:47.603880 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.603865    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fceded2-c12d-4d35-8039-e3fefbc01ed8" containerName="revoke-keys"
Oct 29 22:06:47.604241 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.604217    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fceded2-c12d-4d35-8039-e3fefbc01ed8" containerName="revoke-keys"
Oct 29 22:06:47.608842 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.608822    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.631963 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.628400    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-edc2d3-openshift-default-dockercfg-2pkvj\""
Oct 29 22:06:47.640886 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.640850    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb3e1fca-d76f-4fba-9a59-2cb8600aefda" path="/var/lib/kubelet/pods/eb3e1fca-d76f-4fba-9a59-2cb8600aefda/volumes"
Oct 29 22:06:47.648841 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.648812    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"]
Oct 29 22:06:47.751788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.714744    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.751788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.714836    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.751788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.714943    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.751788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.715101    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.751788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.715208    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b934d8-6393-4cfc-882e-07e965ed9d46-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.751788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.715318    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.751788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.715372    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.751788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.715418    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.751788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.715444    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twjs8\" (UniqueName: \"kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-kube-api-access-twjs8\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.873722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.852273    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.873722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.852405    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.873722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.852507    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.873722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.852683    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.873722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.852784    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b934d8-6393-4cfc-882e-07e965ed9d46-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.873722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.852891    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.873722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.852945    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.873722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.852985    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.873722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.853007    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twjs8\" (UniqueName: \"kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-kube-api-access-twjs8\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.873722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.853820    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.876857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.875618    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.876857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.875912    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.876857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.876517    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b934d8-6393-4cfc-882e-07e965ed9d46-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:47.942344 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.889989    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:48.013653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:47.989195    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:48.213694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:48.213118    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:48.277650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:48.272162    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twjs8\" (UniqueName: \"kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-kube-api-access-twjs8\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:48.277650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:48.273508    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:48.794964 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:48.794874    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"
Oct 29 22:06:48.863332 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:48.846092    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"
Oct 29 22:06:48.867485 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:48.867428    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:49.753197 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:49.753164    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"]
Oct 29 22:06:49.843812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:49.843759    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:06:49.843950 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:49.843856    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:06:49.843950 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:49.843903    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:06:50.508264 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:50.507978    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" event={"ID":"c7b934d8-6393-4cfc-882e-07e965ed9d46","Type":"ContainerStarted","Data":"a97556ac71b2b236481c9226e1ba143d21954e41419390bd19a6ca73e9f7a087"}
Oct 29 22:06:50.508264 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:50.508045    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" event={"ID":"c7b934d8-6393-4cfc-882e-07e965ed9d46","Type":"ContainerStarted","Data":"fbfd343edc056df11ed1e45dc7fba0ae3312e4bcca82c3b4851860cf6f1f0f87"}
Oct 29 22:06:50.814703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:50.793375    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podStartSLOduration=3.793352118 podStartE2EDuration="3.793352118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:50.777969102 +0000 UTC m=+1183.278011402" watchObservedRunningTime="2026-10-29 22:06:50.793352118 +0000 UTC m=+1183.293394418"
Oct 29 22:06:50.887746 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:50.887696    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:51.025648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:51.024045    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:51.492759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:51.492714    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:51.723654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:51.714369    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:06:51.756079 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:51.754953    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:51.773658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:51.763777    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:51.805668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:51.798585    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:51.805668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:51.798685    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:06:52.130668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:52.110709    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:06:52.173079 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:52.171706    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.173079 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:52.171950    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:06:52.310656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:52.310304    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.502708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:52.501783    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.510673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:52.503458    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:06:52.733258 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:52.733213    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.743321 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:52.743291    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:06:53.007804 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:53.007752    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:06:53.265883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:53.174076    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.265883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:53.234224    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:06:53.265883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:53.234331    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:06:53.345650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:53.333638    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.356665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:53.348362    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:06:53.545758 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:53.545665    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.967064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:53.805992    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.210647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:54.202097    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:06:54.225996 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:54.225891    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.355902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:54.280183    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:06:54.355902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:54.353226    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:06:54.467653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:54.420751    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:06:54.546938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:54.546838    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.764058 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:54.754980    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.814710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:54.808073    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.259717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:55.229321    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.261252 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:55.239991    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.315651 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:06:55.301788    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.533s"
Oct 29 22:06:55.388422 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:55.388395    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:06:55.416174 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:55.352171    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.416571 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:55.354096    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.416775 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:55.352136    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.495227 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:55.495175    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.623648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:55.552107    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.698046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:55.693684    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:06:55.698046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:55.693765    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"]
Oct 29 22:06:55.859651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:55.855734    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:06:56.294662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.290734    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:06:56.348655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.345326    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.348655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.345434    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.348655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.345610    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.348655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.345673    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.348655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.345736    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.348655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.345770    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2klk\" (UniqueName: \"kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-kube-api-access-v2klk\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.348655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.345809    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54282f6-acb0-4f9d-9f59-2888caddb88f-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.348655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.345879    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.348655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.345935    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.441655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.413790    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.443731    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.444136    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.469292    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.469365    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.469469    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.469554    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.469775    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.469821    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.469889    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.469923    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2klk\" (UniqueName: \"kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-kube-api-access-v2klk\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.469967    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54282f6-acb0-4f9d-9f59-2888caddb88f-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.471210    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.472466    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.481347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.472804    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.518676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.505162    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:06:56.575798 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.575763    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.598985 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.598953    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.654725 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.599804    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-df42c7-openshift-default-dockercfg-jqslj\""
Oct 29 22:06:56.654928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.600463    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54282f6-acb0-4f9d-9f59-2888caddb88f-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.655059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.633247    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.655807 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.655773    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.737074 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.713752    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:06:56.737074 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.713873    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.795035 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.790248    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.856104 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.856007    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:56.866799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.866764    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:06:56.937507 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:56.918538    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2klk\" (UniqueName: \"kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-kube-api-access-v2klk\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:57.145057 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.144953    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:06:57.208970 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.208935    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"]
Oct 29 22:06:57.209247 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.209230    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"]
Oct 29 22:06:57.230401 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.230372    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"]
Oct 29 22:06:57.230769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.230749    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"
Oct 29 22:06:57.296831 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.295694    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.355848 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.352318    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:06:57.370659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.365238    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-380eea-2zqq7\" (UID: \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"
Oct 29 22:06:57.370659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.365389    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-380eea-2zqq7\" (UID: \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"
Oct 29 22:06:57.370659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.365805    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdxj4\" (UniqueName: \"kubernetes.io/projected/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-kube-api-access-cdxj4\") pod \"maas-api-revoke-keys-e2e-api-a-380eea-2zqq7\" (UID: \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"
Oct 29 22:06:57.380929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.378869    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:06:57.380929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.379057    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:06:57.445718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.419434    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:06:57.483881 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.483833    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:06:57.496801 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.496757    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:06:57.584598 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.565493    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-380eea-2zqq7\" (UID: \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"
Oct 29 22:06:57.598655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.585006    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-380eea-2zqq7\" (UID: \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"
Oct 29 22:06:57.730729 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.715528    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdxj4\" (UniqueName: \"kubernetes.io/projected/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-kube-api-access-cdxj4\") pod \"maas-api-revoke-keys-e2e-api-a-380eea-2zqq7\" (UID: \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"
Oct 29 22:06:57.731076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.616100    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-380eea-2zqq7\" (UID: \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"
Oct 29 22:06:57.756795 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.621907    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:06:57.758029 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.615975    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:06:57.767674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.729453    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.787450 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.739166    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-380eea-2zqq7\" (UID: \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"
Oct 29 22:06:57.847014 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.846971    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.897095 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.897049    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.924377 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.924342    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdxj4\" (UniqueName: \"kubernetes.io/projected/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-kube-api-access-cdxj4\") pod \"maas-api-revoke-keys-e2e-api-a-380eea-2zqq7\" (UID: \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"
Oct 29 22:06:57.961666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.951823    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:06:57.961666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:57.951902    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.094696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:58.092744    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:06:58.094696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:58.092822    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.209691 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:58.202042    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"
Oct 29 22:06:58.238324 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:58.238284    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:06:58.328652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:58.325752    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.356710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:58.336100    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:06:58.420813 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:58.420754    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:06:58.421108 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:58.421075    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.504660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:58.504593    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.519020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:58.518960    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:06:58.862654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:58.774726    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:06:58.862654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:58.804935    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.940759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:58.908739    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.031655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:59.018556    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.447651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:59.410321    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:06:59.447651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:59.424528    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.484520 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:06:59.484426    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:06:59.506826 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:59.506777    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.635701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:59.635399    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.793720 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:59.781759    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.952358 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:59.950592    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:06:59.961144 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:59.961104    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.012997 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:06:59.998461    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.035470 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.034272    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:00.035470 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.034348    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.193697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.193653    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.585138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.585085    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:00.593654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.585776    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:00.603721 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.603672    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:00.614780 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.603519    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" podUID="0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.692898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.692857    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:07:00.757118 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.739346    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.864964 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.422611    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.931126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.931071    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.959017 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.958875    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.967174 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:07:00.967123    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.287s"
Oct 29 22:07:00.967174 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.967176    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"]
Oct 29 22:07:00.967425 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:00.967230    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn" event={"ID":"d54282f6-acb0-4f9d-9f59-2888caddb88f","Type":"ContainerStarted","Data":"99b6bb2c62fdd4fde18cfb09d53b5222767f3720af7cf70efea28349a427d865"}
Oct 29 22:07:01.026791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:01.026746    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:07:01.218786 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:01.218734    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.720806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:01.720758    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.741699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:01.739781    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.870984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:01.855347    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:07:01.933907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:01.933868    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:02.181165 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:02.181113    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:02.181954 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:02.181918    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:07:02.298135 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:02.298086    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Oct 29 22:07:02.464711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:02.449443    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:02.726981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:02.726871    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:02.759875 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:02.759816    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:02.880860 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:02.880798    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:07:02.944699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:02.939778    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.176670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:03.156842    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"]
Oct 29 22:07:03.205104 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:03.205069    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.206432 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:03.206128    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.482788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:03.482716    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.636669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:03.621611    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.676899 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:03.676793    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:03.677217 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:03.677184    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.751656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:03.733842    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.782901 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:03.778652    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.903838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:03.903784    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.954992 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:03.954949    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.243545 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.243441    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.245828 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.244225    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.422660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.417815    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:04.423347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.423108    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.497791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.497749    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.507482 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.507446    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7" event={"ID":"f184d9db-6a8f-4e23-92f5-f36d4108ac1e","Type":"ContainerStarted","Data":"491eb56e0e416b383cef547459e5865a36ba7b20148208393ba1ecfb260d6b34"}
Oct 29 22:07:04.621252 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.621217    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"]
Oct 29 22:07:04.739751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.739709    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.765741 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.744235    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:07:04.769947 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.769571    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:04.769947 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.769753    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.800193 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.800141    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:07:04.896516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.896476    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:04.896818 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.896784    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.910955 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.910914    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.993655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:04.970775    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:07:05.035723 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:05.018823    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:05.035723 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:05.019141    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.067655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:05.038547    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aa11000-z8fmp"]
Oct 29 22:07:05.084836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:05.064580    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": context deadline exceeded"
Oct 29 22:07:05.291445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:05.251869    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:07:05.310571 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:05.256261    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:07:05.380653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:05.375308    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn" event={"ID":"d54282f6-acb0-4f9d-9f59-2888caddb88f","Type":"ContainerStarted","Data":"87ebad69e4423d3dc1a0ebf93e8a1309aa3fb614af9a44879c5657a26ea861bf"}
Oct 29 22:07:05.660184 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:05.660131    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:07:05.718460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:05.703049    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn" podStartSLOduration=12.703022269 podStartE2EDuration="12.703022269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:05.621620007 +0000 UTC m=+1198.121662307" watchObservedRunningTime="2026-10-29 22:07:05.703022269 +0000 UTC m=+1198.203064569"
Oct 29 22:07:05.784754 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:05.784705    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:07:05.813385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:05.813299    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.933939 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:05.919662    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.976817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:05.976770    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:07:06.169817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:06.169767    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:07:06.181920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:06.173700    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn" podUID="d54282f6-acb0-4f9d-9f59-2888caddb88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 29 22:07:06.377863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:06.377803    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.415384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:06.415321    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.809713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:06.806122    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:07:06.837653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:06.829492    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.899187 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:06.879719    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.987470 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:06.985521    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:07:07.017797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.017745    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:07.384510 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.384446    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:07:07.437656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.418680    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:07:07.444660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.443763    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd54e9e8-ebc1-4068-b199-9d0b68200da4" path="/var/lib/kubelet/pods/fd54e9e8-ebc1-4068-b199-9d0b68200da4/volumes"
Oct 29 22:07:07.444660 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:07:07.444389    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.811s"
Oct 29 22:07:07.444660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.444438    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:07:07.464683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.447071    7082 generic.go:356] "Generic (PLEG): container finished" podID="f184d9db-6a8f-4e23-92f5-f36d4108ac1e" containerID="2c8b8a3848e94719bc55cd40292d05e14bb42368623e58bf4cce161e2afbc002" exitCode=0
Oct 29 22:07:07.563566 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.563529    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:07:07.563855 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.563828    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7" event={"ID":"f184d9db-6a8f-4e23-92f5-f36d4108ac1e","Type":"ContainerDied","Data":"2c8b8a3848e94719bc55cd40292d05e14bb42368623e58bf4cce161e2afbc002"}
Oct 29 22:07:07.565413 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.565391    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:07:07.697763 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.622168    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7" podStartSLOduration=12.622148286 podStartE2EDuration="12.622148286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:05.788644739 +0000 UTC m=+1198.288687044" watchObservedRunningTime="2026-10-29 22:07:07.622148286 +0000 UTC m=+1200.122190586"
Oct 29 22:07:08.535644 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:08.512440    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:08.701817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.860192    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:07:08.719653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.860238    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:08.719653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.860261    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded"
Oct 29 22:07:08.719653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:08.010044    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:08.724650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:08.031759    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:08.728651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:08.593585    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:07:09.075997 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:07.722882    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:09.076306 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.076273    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:09.226648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:08.141744    7082 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-wc4l2 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:09.226648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.223025    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2" podUID="4c2dd590-c639-42fe-8a02-4945b6403b3b" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:09.460533 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.460436    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:09.544702 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:07:09.544612    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.921s"
Oct 29 22:07:09.545220 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.545202    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:07:09.545642 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.545609    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:07:09.559371 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.559346    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:07:09.569266 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.569236    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:07:09.607615 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.570799    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"]
Oct 29 22:07:09.607888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.607872    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb7bc281-xvwgv"]
Oct 29 22:07:09.608034 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.608022    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:07:09.668600 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.668559    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb42bae0-7a89-4627-baf2-c467041bacda" path="/var/lib/kubelet/pods/bb42bae0-7a89-4627-baf2-c467041bacda/volumes"
Oct 29 22:07:09.669206 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.669179    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:07:09.695655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.694494    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:07:09.699184 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.699150    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:09.705259 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.705237    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:07:09.805361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.805325    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:07:09.933717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.933683    7082 scope.go:122] "RemoveContainer" containerID="0e609c0a42d4b8da1206ab479d4c042a2271477fb8e031085981a83fac102c03"
Oct 29 22:07:09.965816 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.963760    7082 scope.go:122] "RemoveContainer" containerID="a5b141e06198f90c744b0ae020c887f79fbeb2c039269f8823b1cf0941e00d30"
Oct 29 22:07:09.990590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:09.989832    7082 scope.go:122] "RemoveContainer" containerID="70149604df4e993c107e5d932e728d3429765951206a62b67b09e3b98943f076"
Oct 29 22:07:10.014901 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:10.014362    7082 scope.go:122] "RemoveContainer" containerID="27b48c33af7d9ddb4feb809c37af0ad272c12033d5b65c52268ab83158809847"
Oct 29 22:07:10.066606 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:10.066572    7082 scope.go:122] "RemoveContainer" containerID="3648544a56d4fc16c3f5f186cd49f45e5ee21907d9c21eedac978a1f034465e1"
Oct 29 22:07:10.070540 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:10.070462    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"
Oct 29 22:07:10.111587 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:10.111560    7082 scope.go:122] "RemoveContainer" containerID="b9d349cdb8756357c9999c7bb611cd48516526d8e8ce76d08bfb2e9856a9930c"
Oct 29 22:07:10.248007 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:10.247914    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-xks-ca\") pod \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\" (UID: \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\") "
Oct 29 22:07:10.248917 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:10.248411    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-openshift-ca\") pod \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\" (UID: \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\") "
Oct 29 22:07:10.248917 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:10.248579    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-kube-api-access-cdxj4\" (UniqueName: \"kubernetes.io/projected/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-kube-api-access-cdxj4\") pod \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\" (UID: \"f184d9db-6a8f-4e23-92f5-f36d4108ac1e\") "
Oct 29 22:07:10.249128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:10.248974    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-openshift-ca" pod "f184d9db-6a8f-4e23-92f5-f36d4108ac1e" (UID: "f184d9db-6a8f-4e23-92f5-f36d4108ac1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:07:10.249740 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:10.249713    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:07:10.252519 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:10.252451    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-kube-api-access-cdxj4" pod "f184d9db-6a8f-4e23-92f5-f36d4108ac1e" (UID: "f184d9db-6a8f-4e23-92f5-f36d4108ac1e"). InnerVolumeSpecName "kube-api-access-cdxj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:07:10.270518 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:10.270457    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-xks-ca" pod "f184d9db-6a8f-4e23-92f5-f36d4108ac1e" (UID: "f184d9db-6a8f-4e23-92f5-f36d4108ac1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:07:10.351412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:10.351328    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdxj4\" (UniqueName: \"kubernetes.io/projected/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-kube-api-access-cdxj4\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:07:10.351412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:10.351363    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f184d9db-6a8f-4e23-92f5-f36d4108ac1e-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:07:11.052384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:11.052351    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"
Oct 29 22:07:11.066093 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:11.057481    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7" event={"ID":"f184d9db-6a8f-4e23-92f5-f36d4108ac1e","Type":"ContainerDied","Data":"491eb56e0e416b383cef547459e5865a36ba7b20148208393ba1ecfb260d6b34"}
Oct 29 22:07:11.066279 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:11.066146    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="491eb56e0e416b383cef547459e5865a36ba7b20148208393ba1ecfb260d6b34"
Oct 29 22:07:12.486155 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:12.486120    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"]
Oct 29 22:07:12.674947 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:12.674893    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-751a7416-2nw4w"]
Oct 29 22:07:13.637094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:13.637059    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9097a3-0404-42f8-93c4-1c1f9c7ba79c" path="/var/lib/kubelet/pods/cf9097a3-0404-42f8-93c4-1c1f9c7ba79c/volumes"
Oct 29 22:07:14.046092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:14.046010    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"]
Oct 29 22:07:14.054531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:14.054506    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f4cc552-t24zh"]
Oct 29 22:07:15.636317 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:15.636279    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe98483-acff-4784-8ba6-c9b9ed7e46a3" path="/var/lib/kubelet/pods/fbe98483-acff-4784-8ba6-c9b9ed7e46a3/volumes"
Oct 29 22:07:19.058952 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:19.058916    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"]
Oct 29 22:07:19.068137 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:19.068104    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f83bc8-s59h2"]
Oct 29 22:07:19.642426 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:19.642382    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5333ebc-abe4-4164-a490-a52853830c46" path="/var/lib/kubelet/pods/e5333ebc-abe4-4164-a490-a52853830c46/volumes"
Oct 29 22:07:29.229471 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:07:29.229427    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:07:32.029805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:32.029759    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:32.030314 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:32.029834    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:33.159545 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.159508    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"]
Oct 29 22:07:33.161473 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.161446    7082 cpu_manager.go:414] "containerMap: removing container" podUID="f184d9db-6a8f-4e23-92f5-f36d4108ac1e" containerName="revoke-keys"
Oct 29 22:07:33.161665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.161649    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f184d9db-6a8f-4e23-92f5-f36d4108ac1e" containerName="revoke-keys"
Oct 29 22:07:33.162051 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.162030    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="f184d9db-6a8f-4e23-92f5-f36d4108ac1e" containerName="revoke-keys"
Oct 29 22:07:33.449057 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.448959    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"]
Oct 29 22:07:33.484667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.469794    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"
Oct 29 22:07:33.591818 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.514744    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2d53cd-dockercfg-svwkl\""
Oct 29 22:07:33.592086 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.535992    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d53cd\""
Oct 29 22:07:33.592250 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.592231    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"]
Oct 29 22:07:33.892839 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.892779    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c942cc15-4baf-4eea-bd89-b9b504274bf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds\" (UID: \"c942cc15-4baf-4eea-bd89-b9b504274bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"
Oct 29 22:07:33.893382 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.893338    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fjcs\" (UniqueName: \"kubernetes.io/projected/c942cc15-4baf-4eea-bd89-b9b504274bf9-kube-api-access-8fjcs\") pod \"payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds\" (UID: \"c942cc15-4baf-4eea-bd89-b9b504274bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"
Oct 29 22:07:33.964948 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.964912    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"]
Oct 29 22:07:33.965134 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.965024    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"
Oct 29 22:07:33.995139 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.995108    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fjcs\" (UniqueName: \"kubernetes.io/projected/c942cc15-4baf-4eea-bd89-b9b504274bf9-kube-api-access-8fjcs\") pod \"payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds\" (UID: \"c942cc15-4baf-4eea-bd89-b9b504274bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"
Oct 29 22:07:33.995430 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.995409    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c942cc15-4baf-4eea-bd89-b9b504274bf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds\" (UID: \"c942cc15-4baf-4eea-bd89-b9b504274bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"
Oct 29 22:07:33.996180 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:33.996143    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c942cc15-4baf-4eea-bd89-b9b504274bf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds\" (UID: \"c942cc15-4baf-4eea-bd89-b9b504274bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"
Oct 29 22:07:34.014782 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:34.014749    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:34.097328 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:34.097290    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8402b136-793a-4f5c-a73d-0a4471a9361b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn\" (UID: \"8402b136-793a-4f5c-a73d-0a4471a9361b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"
Oct 29 22:07:34.097752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:34.097710    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk47h\" (UniqueName: \"kubernetes.io/projected/8402b136-793a-4f5c-a73d-0a4471a9361b-kube-api-access-gk47h\") pod \"payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn\" (UID: \"8402b136-793a-4f5c-a73d-0a4471a9361b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"
Oct 29 22:07:34.216725 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:34.208106    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk47h\" (UniqueName: \"kubernetes.io/projected/8402b136-793a-4f5c-a73d-0a4471a9361b-kube-api-access-gk47h\") pod \"payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn\" (UID: \"8402b136-793a-4f5c-a73d-0a4471a9361b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"
Oct 29 22:07:34.216725 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:34.208396    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8402b136-793a-4f5c-a73d-0a4471a9361b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn\" (UID: \"8402b136-793a-4f5c-a73d-0a4471a9361b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"
Oct 29 22:07:34.216725 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:34.211092    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8402b136-793a-4f5c-a73d-0a4471a9361b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn\" (UID: \"8402b136-793a-4f5c-a73d-0a4471a9361b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"
Oct 29 22:07:34.263261 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:34.263184    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fjcs\" (UniqueName: \"kubernetes.io/projected/c942cc15-4baf-4eea-bd89-b9b504274bf9-kube-api-access-8fjcs\") pod \"payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds\" (UID: \"c942cc15-4baf-4eea-bd89-b9b504274bf9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"
Oct 29 22:07:34.270245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:34.267480    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"
Oct 29 22:07:34.299692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:34.299355    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk47h\" (UniqueName: \"kubernetes.io/projected/8402b136-793a-4f5c-a73d-0a4471a9361b-kube-api-access-gk47h\") pod \"payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn\" (UID: \"8402b136-793a-4f5c-a73d-0a4471a9361b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"
Oct 29 22:07:34.399591 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:34.399552    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"
Oct 29 22:07:35.998683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:35.934146    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"]
Oct 29 22:07:35.998683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:35.948822    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"]
Oct 29 22:07:36.786392 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:36.786345    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn" event={"ID":"8402b136-793a-4f5c-a73d-0a4471a9361b","Type":"ContainerStarted","Data":"35755f9a900f5a07f0cdce6559c36e75f9130757f33113a67a6f548b064755b1"}
Oct 29 22:07:36.786738 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:36.786420    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn" event={"ID":"8402b136-793a-4f5c-a73d-0a4471a9361b","Type":"ContainerStarted","Data":"8516ea39e10546ea8fafada5cbb1e298972e37dccf7537b11e90c4f77ce37b60"}
Oct 29 22:07:36.787820 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:36.787748    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"
Oct 29 22:07:36.796542 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:36.795490    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds" event={"ID":"c942cc15-4baf-4eea-bd89-b9b504274bf9","Type":"ContainerStarted","Data":"0bb3771d468f6b5a80d49f8cb4f536b87ee5e5cb8ba8db830ed134213333ae6a"}
Oct 29 22:07:36.796542 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:36.795546    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds" event={"ID":"c942cc15-4baf-4eea-bd89-b9b504274bf9","Type":"ContainerStarted","Data":"19574baab2f65631c5186c428e21dad63ec9e6b96a009abda120c2ac1dd017b0"}
Oct 29 22:07:36.796780 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:36.796652    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"
Oct 29 22:07:36.870738 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:36.870621    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn" podStartSLOduration=3.854846757 podStartE2EDuration="3.854846757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:36.853692062 +0000 UTC m=+1229.353734365" watchObservedRunningTime="2026-10-29 22:07:36.854846757 +0000 UTC m=+1229.354889055"
Oct 29 22:07:36.932860 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:36.932807    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds" podStartSLOduration=4.932793387 podStartE2EDuration="4.932793387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:36.926277247 +0000 UTC m=+1229.426319547" watchObservedRunningTime="2026-10-29 22:07:36.932793387 +0000 UTC m=+1229.432835686"
Oct 29 22:07:39.761964 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:39.761909    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"]
Oct 29 22:07:39.772285 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:39.772262    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:39.865298 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:39.865262    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"]
Oct 29 22:07:39.952931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:39.946257    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:39.952931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:39.946405    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:39.952931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:39.946524    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljcbn\" (UniqueName: \"kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-kube-api-access-ljcbn\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:39.952931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:39.946555    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:39.952931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:39.946643    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:39.952931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:39.946714    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:39.952931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:39.946752    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:39.952931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:39.946856    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:39.952931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:39.946878    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.048073 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.047977    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.048277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.048077    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.048277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.048145    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljcbn\" (UniqueName: \"kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-kube-api-access-ljcbn\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.048277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.048165    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.048277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.048206    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.048277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.048247    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.048277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.048271    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.048649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.048332    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.048649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.048347    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.048649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.048560    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.048831 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.048656    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.048831 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.048816    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.049729 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.049676    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.049966 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.049941    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.052431 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.052315    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.052431 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.052393    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.072674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.072645    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.072777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.072704    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljcbn\" (UniqueName: \"kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-kube-api-access-ljcbn\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.171013 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.170954    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:40.938003 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.937970    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"]
Oct 29 22:07:40.949877 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.949527    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:40.949877 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.949719    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:40.949877 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.949778    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:40.981230 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:40.979323    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" event={"ID":"192f4565-6fc9-4fde-9f70-bc9da0b26a07","Type":"ContainerStarted","Data":"ed94eb1b15bb06b4a2450ae114a7185ae318089a6bbf2029bdca00adc36a9088"}
Oct 29 22:07:42.026198 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:42.026162    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" event={"ID":"192f4565-6fc9-4fde-9f70-bc9da0b26a07","Type":"ContainerStarted","Data":"409b522d598d1392f782b1e8d7576d9f96260387cdca9991c67083b6e5389ea5"}
Oct 29 22:07:42.110169 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:42.110070    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" podStartSLOduration=3.110048818 podStartE2EDuration="3.110048818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:42.108134957 +0000 UTC m=+1234.608177256" watchObservedRunningTime="2026-10-29 22:07:42.110048818 +0000 UTC m=+1234.610091117"
Oct 29 22:07:42.171388 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:42.171349    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:42.173481 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:42.173453    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 29 22:07:42.173640 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:42.173519    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" podUID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 29 22:07:43.207652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.204545    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 29 22:07:43.207652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.204642    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" podUID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 29 22:07:43.609654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.603409    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"]
Oct 29 22:07:43.609654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.607672    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:43.609654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.607764    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:43.679427 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.679390    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.879461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.872933    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"]
Oct 29 22:07:43.879461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.873406    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.879461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.873645    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.879461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.873843    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.879461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.873878    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.879461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.873903    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626zg\" (UniqueName: \"kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-kube-api-access-626zg\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.879461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.873956    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.879461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.874008    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6e6557-f68f-423d-bc71-67dcb9f25587-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.879461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.874096    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.879461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.874158    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.893958 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.893928    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"]
Oct 29 22:07:43.894200 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.894182    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"]
Oct 29 22:07:43.894418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.894401    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:43.976692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.976286    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.978338 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.977246    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.978338 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.977309    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.978338 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.977576    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626zg\" (UniqueName: \"kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-kube-api-access-626zg\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.978338 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.977670    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.978338 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.977723    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6e6557-f68f-423d-bc71-67dcb9f25587-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.978338 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.977804    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.978338 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.977860    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.978338 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.977957    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.978700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.978487    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6e6557-f68f-423d-bc71-67dcb9f25587-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.979221 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.979199    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.979450 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.979434    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.979873 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.979826    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:43.980351 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:43.980318    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:44.053283 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.048889    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:44.053283 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.053192    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:44.079511 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.079435    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:44.080208 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.080170    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.080360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.080268    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.080360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.080329    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.080493 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.080384    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.080552 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.080536    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.080843 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.080820    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.081054 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.081023    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.081225 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.081193    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.081508 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.081455    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qllq4\" (UniqueName: \"kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-kube-api-access-qllq4\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.083496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.083034    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626zg\" (UniqueName: \"kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-kube-api-access-626zg\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:44.105212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.093661    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:44.231417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.203816    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.231417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.203987    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.231417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.204020    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.231417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.204048    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.231417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.204186    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qllq4\" (UniqueName: \"kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-kube-api-access-qllq4\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.231417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.204318    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.231417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.204393    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.231417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.204460    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.231417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.204529    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.231417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.205265    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.231417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.205535    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.306393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.238646    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.306393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.239153    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.306393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.250174    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.306393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.254853    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.306393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.268340    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 29 22:07:44.306393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.268397    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" podUID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 29 22:07:44.449561 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.449529    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.450451 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.450427    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.458363 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.457458    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qllq4\" (UniqueName: \"kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-kube-api-access-qllq4\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:44.717661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:44.717557    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:45.172428 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:45.172338    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 29 22:07:45.172428 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:45.172399    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" podUID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 29 22:07:45.775656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:45.775550    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"]
Oct 29 22:07:45.780496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:45.780471    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"]
Oct 29 22:07:45.931668 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:07:45.931126    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6e6557_f68f_423d_bc71_67dcb9f25587.slice/crio-e7f6bc7e916ec5cd27edd31333b3720a96f0bdfd664222dc05b83dae1c764f81 WatchSource:0}: Error finding container e7f6bc7e916ec5cd27edd31333b3720a96f0bdfd664222dc05b83dae1c764f81: Status 404 returned error can't find the container with id e7f6bc7e916ec5cd27edd31333b3720a96f0bdfd664222dc05b83dae1c764f81
Oct 29 22:07:45.935966 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:45.935099    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:45.935966 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:45.935190    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:45.935966 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:45.935233    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:45.942228 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:45.941571    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:45.942228 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:45.941719    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:45.942228 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:45.941767    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:46.172519 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:46.172490    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 29 22:07:46.172720 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:46.172544    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" podUID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 29 22:07:46.783614 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:46.783541    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" event={"ID":"db6e6557-f68f-423d-bc71-67dcb9f25587","Type":"ContainerStarted","Data":"e7f6bc7e916ec5cd27edd31333b3720a96f0bdfd664222dc05b83dae1c764f81"}
Oct 29 22:07:46.822509 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:46.822384    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" event={"ID":"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019","Type":"ContainerStarted","Data":"0d8dacb6696d6abbfbe9100db97609a0d09131922c3ba13c0073c49596c4a783"}
Oct 29 22:07:47.015726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:47.009722    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:47.015726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:47.009788    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:47.191445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:47.180781    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 29 22:07:47.191445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:47.180857    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" podUID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 29 22:07:48.115605 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:47.818323    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"
Oct 29 22:07:48.115605 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:47.947939    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" event={"ID":"db6e6557-f68f-423d-bc71-67dcb9f25587","Type":"ContainerStarted","Data":"0d7c44b569eed8da940180306301764490f92e94f88d4556487bb14744964f45"}
Oct 29 22:07:48.115605 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:47.954344    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" event={"ID":"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019","Type":"ContainerStarted","Data":"49e6a240c06c3e7a6accf704fccfaac7ff3690fc731a997b60d5b5a3eee8792e"}
Oct 29 22:07:48.115605 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:48.027103    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" podStartSLOduration=5.027075842 podStartE2EDuration="5.027075842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:48.007531172 +0000 UTC m=+1240.507573473" watchObservedRunningTime="2026-10-29 22:07:48.027075842 +0000 UTC m=+1240.527118142"
Oct 29 22:07:48.115605 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:48.105686    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:48.115605 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:48.105819    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 22:07:48.176646 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:48.170739    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 22:07:48.223320 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:48.217679    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 29 22:07:48.223320 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:48.217738    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" podUID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 29 22:07:48.269543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:48.268586    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" podStartSLOduration=5.268566143 podStartE2EDuration="5.268566143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:48.218219628 +0000 UTC m=+1240.718261928" watchObservedRunningTime="2026-10-29 22:07:48.268566143 +0000 UTC m=+1240.768608442"
Oct 29 22:07:48.720609 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:48.720512    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:48.724861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:48.724276    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:07:48.724861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:48.724334    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 29 22:07:48.987252 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:48.987190    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"
Oct 29 22:07:49.096423 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:49.096212    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 22:07:49.096423 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:49.096270    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 22:07:49.171591 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:49.171560    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 29 22:07:49.177403 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:49.172252    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" podUID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 29 22:07:49.354077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:49.353913    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" event={"ID":"93b8ea59-e14b-4658-abea-a9234b68fbb9","Type":"ContainerDied","Data":"867962ef794ba40d2870c2150b83fdac0911ec4e1caae3f4fac181e863f56027"}
Oct 29 22:07:49.417654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:49.403349    7082 scope.go:122] "RemoveContainer" containerID="867962ef794ba40d2870c2150b83fdac0911ec4e1caae3f4fac181e863f56027"
Oct 29 22:07:49.417654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:49.324730    7082 generic.go:356] "Generic (PLEG): container finished" podID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerID="867962ef794ba40d2870c2150b83fdac0911ec4e1caae3f4fac181e863f56027" exitCode=0
Oct 29 22:07:49.523406 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:49.522918    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:49.523406 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:49.523032    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:49.523406 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:49.523077    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:07:49.730865 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:49.730770    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:07:49.731211 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:49.731174    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 29 22:07:50.110651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:50.097040    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 22:07:50.110651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:50.097100    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 22:07:50.180953 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:50.180915    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:50.182801 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:50.182779    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:50.183760 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:50.183714    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:07:51.331450 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.331409    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:07:51.436590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.431803    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:51.436590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.431860    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8" podUID="423fc4e9-6575-4690-9263-83ffff40dfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:51.477908 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.470251    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:51.499650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.478706    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:07:51.499650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.479499    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:07:51.499650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.479542    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 29 22:07:51.499650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.494773    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:51.588707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.588184    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:51.754121 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.624716    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:07:51.811261 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.796515    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:07:51.811261 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.796572    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 29 22:07:51.819167 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.819116    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q" podUID="86f63896-629a-4327-a138-279748c035bd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.60:50052\" within 1s: context deadline exceeded"
Oct 29 22:07:51.863944 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.849600    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 22:07:51.863944 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:51.849779    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 22:07:51.976654 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:07:51.954246    7082 logging.go:55] [core] [Channel #1771 SubChannel #1772]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.60:50052", ServerName: "10.134.0.60:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:39868->10.134.0.60:50052: use of closed network connection"
Oct 29 22:07:52.082647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.076345    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:52.145808 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.128779    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 22:07:52.145808 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.128838    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 22:07:52.231915 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.231671    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Oct 29 22:07:52.501356 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.501254    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:52.545780 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.545718    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:52.556656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.547664    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:07:52.609001 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.568286    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:07:52.609001 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.599697    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:07:52.652043 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:07:52.652001    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.01s"
Oct 29 22:07:52.652245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.652212    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:07:52.652315 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.652278    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:07:52.652374 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.652344    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:07:52.652460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.652406    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:07:52.652524 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.652465    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:07:52.653278 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.653242    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" event={"ID":"93b8ea59-e14b-4658-abea-a9234b68fbb9","Type":"ContainerStarted","Data":"0cbb201369202d83e2353e9943ce27aba7470bf0bed95613c1e700486fa0d22b"}
Oct 29 22:07:52.686081 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.677173    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"]
Oct 29 22:07:52.710753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.706759    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:52.947651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.930001    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:07:52.947651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:52.930086    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 29 22:07:53.079555 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:53.079524    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:07:53.131962 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:53.131892    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f6171d-mjqnk"]
Oct 29 22:07:53.133262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:53.099955    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 22:07:53.141750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:53.141705    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 22:07:53.568825 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:53.568778    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:53.682716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:53.682351    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:07:53.747549 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:53.747505    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:07:53.749227 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:53.749194    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:07:53.761706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:53.761657    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:53.866889 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:53.866795    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:07:53.867168 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:53.867132    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 29 22:07:53.896796 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:53.896748    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:54.097068 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:54.093208    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:54.678060 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:54.678021    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:07:54.687793 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:54.687738    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:07:54.715702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:54.712711    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:54.739688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:54.730967    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:07:54.739688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:54.735712    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:54.834511 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:54.834466    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:07:54.865795 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:54.865746    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 29 22:07:54.901043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:54.896384    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:07:54.909064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:54.909025    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:07:54.925767 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:54.925726    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" podUID="0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:07:55.068789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.013360    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:55.068789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.047836    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:07:55.180936 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.179753    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:07:55.180936 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.179820    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:07:55.180936 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.180615    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": context deadline exceeded"
Oct 29 22:07:55.224803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.191252    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:55.224803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.191304    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:55.224803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.191589    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 22:07:55.224803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.191621    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 22:07:55.320713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.310554    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": context deadline exceeded"
Oct 29 22:07:55.354123 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.354076    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fceded2-c12d-4d35-8039-e3fefbc01ed8" path="/var/lib/kubelet/pods/3fceded2-c12d-4d35-8039-e3fefbc01ed8/volumes"
Oct 29 22:07:55.366748 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:07:55.366711    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.662s"
Oct 29 22:07:55.390520 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.390489    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:07:55.719265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.718815    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:07:55.720112 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.720086    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:07:55.720307 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.720286    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 29 22:07:55.745133 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.745093    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:55.933426 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.902756    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:55.949654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:55.912321    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:56.013291 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:56.013196    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:07:56.015692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:56.015653    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:07:56.017793 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:56.016994    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:56.295848 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:56.282447    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:56.371767 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:56.360594    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 22:07:56.371767 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:56.360677    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 22:07:56.372180 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:56.372139    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:07:56.721654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:56.720726    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:56.722167 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:56.721933    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:07:56.722167 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:56.721980    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 29 22:07:56.768760 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:56.768718    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:07:56.835653 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:07:56.832922    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.21s"
Oct 29 22:07:56.958652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:56.940798    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:56.974911 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:56.974798    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:07:57.057652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.032340    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:57.060817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.060784    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:07:57.086056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.086015    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:07:57.142652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.131742    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 22:07:57.142652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.131802    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 22:07:57.282818 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.282716    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:57.292297 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.292267    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"]
Oct 29 22:07:57.381172 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.379645    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:57.411656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.408837    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"]
Oct 29 22:07:57.411656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.408936    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:07:57.411656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.410170    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"
Oct 29 22:07:57.532652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.522435    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnvbm\" (UniqueName: \"kubernetes.io/projected/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-kube-api-access-lnvbm\") pod \"maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n\" (UID: \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"
Oct 29 22:07:57.551020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.533313    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:07:57.551020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.533536    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:07:57.551020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.534357    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:07:57.551020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.534464    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:07:57.620695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.602912    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n\" (UID: \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"
Oct 29 22:07:57.620695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.603202    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n\" (UID: \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"
Oct 29 22:07:57.781005 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.780943    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:07:57.781690 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.781644    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 29 22:07:57.871926 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.871826    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:57.874179 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.874144    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:07:57.880660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.880613    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnvbm\" (UniqueName: \"kubernetes.io/projected/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-kube-api-access-lnvbm\") pod \"maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n\" (UID: \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"
Oct 29 22:07:57.881185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.881022    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n\" (UID: \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"
Oct 29 22:07:57.881471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.881447    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n\" (UID: \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"
Oct 29 22:07:57.884184 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.884144    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n\" (UID: \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"
Oct 29 22:07:58.022652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:57.988145    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:58.419616 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.419427    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 29 22:07:58.419616 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.419501    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 29 22:07:58.510035 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.493054    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n\" (UID: \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"
Oct 29 22:07:58.510035 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.493688    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:58.602919 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.600712    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:58.602919 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.600776    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:58.602919 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.601008    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:07:58.602919 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.601036    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:07:58.602919 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.602215    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:07:58.684305 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.684199    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:07:58.733858 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.733812    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:58.841942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.841810    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:07:58.841942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.841872    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 29 22:07:58.981712 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:07:58.981310    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.358s"
Oct 29 22:07:58.981712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.981389    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:07:58.981712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.981488    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:07:58.981712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.981517    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:07:58.988655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.984690    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnvbm\" (UniqueName: \"kubernetes.io/projected/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-kube-api-access-lnvbm\") pod \"maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n\" (UID: \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"
Oct 29 22:07:58.996694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:58.990310    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:07:59.025651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:59.021125    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:07:59.025651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:59.021180    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:07:59.147655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:59.141861    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:59.174253 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:07:59.161194    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:07:59.174253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:59.162219    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"
Oct 29 22:07:59.245660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:59.214003    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:07:59.619134 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:59.545910    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:59.665821 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:59.648727    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:59.724705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:59.724674    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:07:59.725006 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:59.724975    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:07:59.725140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:59.725129    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:59.766095 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:59.766068    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:07:59.770062 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:59.769172    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:07:59.877816 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:07:59.876470    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:08:00.171638 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:00.171252    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:08:00.171638 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:00.171334    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:08:00.172460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:00.172412    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:08:00.213650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:00.182990    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:08:01.000287 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:00.985672    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:08:01.040545 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:01.039033    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:08:01.040545 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:01.039099    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:08:01.264137 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:01.264039    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"]
Oct 29 22:08:01.304652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:01.275546    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2" podUID="43619c62-7adb-49b0-94c8-870ec330cd19" containerName="istio-proxy" containerID="cri-o://00d6c71b54c18baf5f3c04df568e9ac647fd849c24c10f71a5e19b2d308cac46" gracePeriod=30
Oct 29 22:08:01.739655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:01.721227    7082 generic.go:356] "Generic (PLEG): container finished" podID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerID="a8703b676a08da768e9ed481426b8ca9ef2e64ac4db335f3e4d562eacf708558" exitCode=0
Oct 29 22:08:01.770085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:01.769107    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"]
Oct 29 22:08:01.770085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:01.769194    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" event={"ID":"5f7332e1-ce42-4071-bf1a-ddf706996684","Type":"ContainerDied","Data":"a8703b676a08da768e9ed481426b8ca9ef2e64ac4db335f3e4d562eacf708558"}
Oct 29 22:08:01.770085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:01.769786    7082 scope.go:122] "RemoveContainer" containerID="a8703b676a08da768e9ed481426b8ca9ef2e64ac4db335f3e4d562eacf708558"
Oct 29 22:08:01.773975 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:01.773812    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:08:01.775509 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:01.774134    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:08:01.775509 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:01.774254    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:08:02.068745 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:02.064446    7082 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 29 22:08:02.068745 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:02.064506    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 29 22:08:02.747680 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:02.747547    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n" event={"ID":"9ce8bea9-8a70-4e28-9d56-53a0932d1b93","Type":"ContainerStarted","Data":"2f5fba3b6562539cf123efac2b9e582d85c723e4a8354e3782217909aa0c16f0"}
Oct 29 22:08:02.747680 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:02.747610    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n" event={"ID":"9ce8bea9-8a70-4e28-9d56-53a0932d1b93","Type":"ContainerStarted","Data":"f54bc0408221e600a03c63d05bd17d1c001aec126a76f66c1726ab39a427731c"}
Oct 29 22:08:02.750972 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:02.750943    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" event={"ID":"5f7332e1-ce42-4071-bf1a-ddf706996684","Type":"ContainerStarted","Data":"f0f462309de1e237e69e18727b7d47a0c7131b2f039708f08a868f8210cf6253"}
Oct 29 22:08:02.798543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:02.798482    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n" podStartSLOduration=7.798462264 podStartE2EDuration="7.798462264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:02.793266165 +0000 UTC m=+1255.293308466" watchObservedRunningTime="2026-10-29 22:08:02.798462264 +0000 UTC m=+1255.298504564"
Oct 29 22:08:02.903290 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:02.903207    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:08:03.118355 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:03.118319    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:08:03.122974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:03.122926    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:08:03.839307 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:03.838885    7082 generic.go:356] "Generic (PLEG): container finished" podID="9ce8bea9-8a70-4e28-9d56-53a0932d1b93" containerID="2f5fba3b6562539cf123efac2b9e582d85c723e4a8354e3782217909aa0c16f0" exitCode=0
Oct 29 22:08:03.839307 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:03.839007    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n" event={"ID":"9ce8bea9-8a70-4e28-9d56-53a0932d1b93","Type":"ContainerDied","Data":"2f5fba3b6562539cf123efac2b9e582d85c723e4a8354e3782217909aa0c16f0"}
Oct 29 22:08:04.335964 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:04.210690    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 29 22:08:04.347245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:04.347203    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 29 22:08:04.928773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:04.916016    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 29 22:08:04.928773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:04.916083    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 29 22:08:05.916787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:05.916298    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 29 22:08:05.916787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:05.916362    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 29 22:08:06.032663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:06.030834    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:08:06.032663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:06.030899    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8" podUID="423fc4e9-6575-4690-9263-83ffff40dfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:06.922766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:06.906248    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 29 22:08:07.019905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:06.928283    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"]
Oct 29 22:08:07.019905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:06.928967    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"]
Oct 29 22:08:07.019905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:06.929156    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"]
Oct 29 22:08:07.019905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:06.944091    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 29 22:08:07.019905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:06.944178    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" podUID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" containerName="istio-proxy" containerID="cri-o://800159fb2e9afbc67db3c9c6b50fec50328770c8edbf2efd22113a417c37e2df" gracePeriod=30
Oct 29 22:08:07.019905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:06.944342    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy" containerID="cri-o://d82bec39cf4f34f2d7688641833b8b0044ac07fd0cfe0d375d516046a66ddf87" gracePeriod=30
Oct 29 22:08:07.019905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:06.944436    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy" containerID="cri-o://efee2de6f820641c7f87224c2c71d0a7984816e3e82ddfcec7b2ccc1c2b84800" gracePeriod=30
Oct 29 22:08:07.444893 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:07.053687    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"]
Oct 29 22:08:07.444893 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:07.062816    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d14b5e7c-p2cmw"]
Oct 29 22:08:07.895811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:07.880746    7082 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-wc4l2 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded" start-of-body=
Oct 29 22:08:07.895811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:07.880810    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2" podUID="4c2dd590-c639-42fe-8a02-4945b6403b3b" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded"
Oct 29 22:08:08.881235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:08.881176    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:08:08.881486 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:08.881321    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:08.939840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:08.881685    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:08.939840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:08.881793    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:08:08.939840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:08.881901    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:08:08.939840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:08.881951    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:08:08.939840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:08.908738    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:08:08.939840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:08.908825    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:10.106857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:09.009747    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt" podUID="f069baec-59d9-4c9e-b5a2-c25261d01cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:10.106857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:09.089683    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:10.106857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:09.182303    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" podUID="4684c6fd-8ac7-4005-a20d-48396c0ab981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:10.106857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:09.209509    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs" podUID="c32b079b-eba6-4e6f-8f79-f9c3cb86de99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:10.106857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:09.589493    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" podUID="0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:10.106857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:09.881835    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:10.106857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:09.881957    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:10.106857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:09.882017    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:10.106857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:09.883117    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:08:10.106857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:09.909541    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:08:10.106857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:09.909603    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:09.881797    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.372598    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.372690    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.386479    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs" podUID="c32b079b-eba6-4e6f-8f79-f9c3cb86de99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.387826    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.387861    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.393988    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs" podUID="f4b42b80-df94-4d12-80cb-5814f733d918" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.29:8081/healthz\": dial tcp 10.134.0.29:8081: connect: connection refused"
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.395174    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.395209    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.399568    7082 patch_prober.go:29] interesting pod/dns-default-245p6 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.399594    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-245p6" podUID="3accc688-8a04-4462-9593-228a18548478" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.417956    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.418025    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" podUID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.431231    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs" podUID="f4b42b80-df94-4d12-80cb-5814f733d918" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.29:8081/readyz\": dial tcp 10.134.0.29:8081: connect: connection refused"
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:08:22.441407    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a92f63_b5c0_4f00_9976_5bbc789b2bed.slice/crio-d82bec39cf4f34f2d7688641833b8b0044ac07fd0cfe0d375d516046a66ddf87.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a92f63_b5c0_4f00_9976_5bbc789b2bed.slice/crio-conmon-d82bec39cf4f34f2d7688641833b8b0044ac07fd0cfe0d375d516046a66ddf87.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b822e3a_e780_4b65_98bc_f0c26990bfcf.slice/crio-800159fb2e9afbc67db3c9c6b50fec50328770c8edbf2efd22113a417c37e2df.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b42b80_df94_4d12_80cb_5814f733d918.slice/crio-1b1ce3527cb87d2cb6197f058b48c9bd38090af9b9c969e48a2778ec15377f45.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.446732    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs" podUID="f4b42b80-df94-4d12-80cb-5814f733d918" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.29:8081/readyz\": dial tcp 10.134.0.29:8081: connect: connection refused"
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.456944    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x" podUID="79e8c570-3e0f-40b7-9696-81b84b10e8eb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.72:50052\" within 1s: context deadline exceeded"
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.458095    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.537395834s: [/var/lib/containers/storage/overlay/5278faf375b8f61dd54419452b12bc91aa1600598d40b13111edf0e33123f518/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.458246    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.537554865s: [/var/lib/containers/storage/overlay/883de5e6a5ee14854d4d632d25caad5784481a5b76b506ccc47f2585151b041b/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.458383    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.537689829s: [/var/lib/containers/storage/overlay/50ff2f939a28029cded3bea08a10d23578da3d9db1d227b5cf7dfeea86561f14/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.458521    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.537797246s: [/var/lib/containers/storage/overlay/f8f56aa9dd184171f3e012e4d3089d4bc797154600545c073b0e089bd3762665/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.458545    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.537819425s: [/var/lib/containers/storage/overlay/97d6aff82409c02b8d792fca3f10c557b5364903eb344ae8267ebfa3562e37c4/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.458858    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.538126146s: [/var/lib/containers/storage/overlay/28d55dd6f6fb59ae12a6916ba4004ec2a08667869c165f2d9bcc59ff70cf9188/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.458889    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.532034562s: [/var/lib/containers/storage/overlay/a22381ee08c46089aa6f990fdd5f64e57c3565f2aeb271e9ea723322e2ace4fe/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.458913    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.532056388s: [/var/lib/containers/storage/overlay/aceb5e55de3950b68e941c8f98706eb760e9ebec086e9d9572f9b181dd0fc931/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459002    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.490258224s: [/var/lib/containers/storage/overlay/c4b852c70d25182306a9c042941f5ca537aa6500991c796fdb6970f07fe48f69/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459050    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.457262357s: [/var/lib/containers/storage/overlay/d578b1a5bbf58f0ed96299ecc48104a759521bcd1377d9e24af9e6bada2c1cec/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459053    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.457264012s: [/var/lib/containers/storage/overlay/28d75aaf44f7bd52e79443d127c979b13d56c9ec1227d11c5d939f48afa1b2b8/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459075    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.457284239s: [/var/lib/containers/storage/overlay/3e615e674da6541b268cfb144ae6cb31bfdfd396157ec4fb6e2caaafae08822a/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459100    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.457304999s: [/var/lib/containers/storage/overlay/7f5afc93218b9391b2df4d5b1e0738a5d54d21b7173ceca4ad33fa492b03d8ac/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459123    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.457327072s: [/var/lib/containers/storage/overlay/a23203f42c7964f04162d80fdc860c0c684fdd51e68bb58de0da28676d5c3c4f/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459147    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.457349581s: [/var/lib/containers/storage/overlay/cd26609ceec037a07d261163f8d53fe9af5c9279bdae63ecfa24be34ba7ef5cf/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459170    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.45737124s: [/var/lib/containers/storage/overlay/a28938d829e7ea4e7151fa8d64813887f26337bb5a2e0a20f3b7318569f31a52/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459266    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.457446704s: [/var/lib/containers/storage/overlay/e88cbb29a0d6de72bf322ce550431947b2c44758f4508df27a6f10748f0ebe06/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459291    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.457471077s: [/var/lib/containers/storage/overlay/17eb54733ecb91122bb2b966e14725717b21e882aabe3ca16ed6af2b4bb201d3/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459310    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.427332146s: [/var/lib/containers/storage/overlay/0b0aab0464cd320e5626d8cacae6d891a874331e27de231be6b3b1e0221049d7/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459318    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.457496485s: [/var/lib/containers/storage/overlay/cc75434104519032dd3349d7e363bb948e59d887aed6806b2f150513bc72df8b/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459349    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.426320658s: [/var/lib/containers/storage/overlay/63099bed5b1b3ee6df81b86a68811bd8146788476eec755a56ad202540c26e4e/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459375    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.426344163s: [/var/lib/containers/storage/overlay/bc93e8c32926c9280ec2b03952cb31d1e85e1e3eaa5b35202c4fb3ed2b34953c/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459398    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.426365899s: [/var/lib/containers/storage/overlay/da9640e2b5140fb09301f87ca38db98c43bf5063f5be0f64b6de62d7aaf95220/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459420    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.426386786s: [/var/lib/containers/storage/overlay/efab40575f5df9e2dfdf9e8fde7339a20429561474ced6459e6691f33422bd6b/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459445    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.425299632s: [/var/lib/containers/storage/overlay/fbb76771e1b306f1a08ef56a70158847448f45fb9a64a7359d06d651314e1535/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459504    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.425355179s: [/var/lib/containers/storage/overlay/e11fbccbb1abe8b466973e34ee297ac18f4ace7f1ed9637f15f3e3a42a4bb44e/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459528    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.425377709s: [/var/lib/containers/storage/overlay/7006d0649d5552f21b2e3979ff8b116ebab881febb7a5a6238b426bedf405a91/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459551    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.425399686s: [/var/lib/containers/storage/overlay/1c235ec1a2a9f60a443c68779f8a3030d319381915d6dd66595db305d981f578/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459576    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.425422673s: [/var/lib/containers/storage/overlay/a31773804693e801f2d1061806ab26c933e678fd0d1240ffa5d47b10b290209c/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459750    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.425590762s: [/var/lib/containers/storage/overlay/30e9e3bc7e9b5332d7f6e71813d03d83fe9ecb216aa0dd217524ded0a122f82b/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459775    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.425614989s: [/var/lib/containers/storage/overlay/608f30a85ecb952cda557a83e7c1870aa1343d7e485fb950e59d74e53ce20bd5/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459800    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.390798223s: [/var/lib/containers/storage/overlay/4aade99f389fb84973d6232bfe5d718914136ec2e5e97c58eac3401fea0950bf/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459824    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.390818366s: [/var/lib/containers/storage/overlay/de5a25e6c550396dd77e5924b7a3299f9fd1b3c70d30a6b018cfe26fb40251c4/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459848    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.390839837s: [/var/lib/containers/storage/overlay/f236da616fad0d883c550dd9ae6edf62de4aac06683ad92a3e7748456e21eb11/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459938    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.390900138s: [/var/lib/containers/storage/overlay/0e2ddd55ea565b4d6132c8c4a4333cf234d993a94a5c5569aaca6ba0898472ec/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459962    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.390922905s: [/var/lib/containers/storage/overlay/ae2c4fece1cab5dcc4d9d4188e1e80c6610e86fb3f3e07ab3a8235e9470449f4/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459966    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.390927638s: [/var/lib/containers/storage/overlay/ff86e2d515917fb7b870d21404561941e80b3795ae56b9594d88b7396bbe5795/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.459991    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.390944557s: [/var/lib/containers/storage/overlay/e80fae41a67db8ca5ec99b5f3637921f39921e368f2a46674ce322b62d2fed42/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460001    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.390943072s: [/var/lib/containers/storage/overlay/4402fb530ea19e09777f5a8923e18195319945138f202416f2584f604cf8bc46/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460013    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.390946303s: [/var/lib/containers/storage/overlay/2f90980733d5946852c3a69987819b0f7fc6dfa72659e42965bf029e38d58a0a/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460036    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.390988286s: [/var/lib/containers/storage/overlay/56cad2dec67bd57a2bdde7bec62d78cded0b9950088beeef31333196a3ba087e/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460038    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.390969736s: [/var/lib/containers/storage/overlay/d7600ba39efb0f7a69ad8625fe2b03897ad4bbe1f13136b62b79326882f5b774/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460077    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.389926963s: [/var/lib/containers/storage/overlay/c2ca883e49c5b818a115e192a76b6135e4fa27020665d553c4f73a81672f4fb1/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460138    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.389985333s: [/var/lib/containers/storage/overlay/acfc72190ab96f5ead9c1bf2e4d697acc28af6a6aa51286e64dee371dc5f986a/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460162    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.387876308s: [/var/lib/containers/storage/overlay/bdb0e627a21f58fc411e801d2294f0fe514f6ad632690beee144fbc937ed6ec8/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460185    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.387897651s: [/var/lib/containers/storage/overlay/e00ed6f7e3246eb1ba5f2e75ff47321f91fed1fc93d6a4b6d4b36410abc54966/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460231    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.386868061s: [/var/lib/containers/storage/overlay/3d884b6f9762b941697c5d8234d002fc6c4093c80f0ae29372e149bc7bdf59dc/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460262    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.386896964s: [/var/lib/containers/storage/overlay/96a66500e734d320570f1f54179dda34c8701eda78a1f8031bea03725ff16a01/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460290    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.386923896s: [/var/lib/containers/storage/overlay/c740a1e8f86e2d9f504786bd5de7d4d99b7faeb617c2860977377a729415a934/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460547    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.380793943s: [/var/lib/containers/storage/overlay/e3d40403b97965045745e62613acc8604f3923c8dec870c80188cbaf319811d7/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460734    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.380953497s: [/var/lib/containers/storage/overlay/4632d4ef506b54fa02ead4a7d4263263fa0bd826e02c6f6b1c2f74aec7574b87/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.460943    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.349237858s: [/var/lib/containers/storage/overlay/395ec44032f954937668f13f41d78f0ea2453ffd9354629a2541dcd453ed36d5/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461027    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.349320711s: [/var/lib/containers/storage/overlay/c96f87e8dfdb4eb8ca49fa22db98ef6b3ed1202bdfac1a3f37dc71c8477210e3/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461061    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.349353124s: [/var/lib/containers/storage/overlay/d5a8c0e91d50baebaec0828f2b45519f21bc7fa4ab991149e754a14f4cfb201f/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461209    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.205058109s: [/var/lib/containers/storage/overlay/c2f8bb569f87bd48343e16d672873f89f64253d208619fa865193c66b6d39efa/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461451    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.169590339s: [/var/lib/containers/storage/overlay/c8367b4e3ff64441c666fae99945acc87d1eb7d4fdd3bfa527962a1a1ae5db65/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461478    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.009116796s: [/var/lib/containers/storage/overlay/9293fc1642d72ba76d3cb2accf980c68886577568d6c004e46cd27400f4fca9b/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461512    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.009145959s: [/var/lib/containers/storage/overlay/8326431a2a49c5f34d0ee423fe35ef9d28000f200d771795857836d896275b82/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461520    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.996040888s: [/var/lib/containers/storage/overlay/3620037d67fff06a4400353c08e98bf3b8e2f14abb3e78230537ce724ffb409c/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461537    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.949652951s: [/var/lib/containers/storage/overlay/8d64493d63f1d912a6b9159423877b2ff5353e69efec6bb66fb657962e064143/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461549    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.949643616s: [/var/lib/containers/storage/overlay/0c108d75e8a83eeff11c57d31972dd29b83a3378b4cd8cfbbb25295b37e074ea/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461562    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.949654233s: [/var/lib/containers/storage/overlay/8741a39a05fdbd31829e8e9bfb64e44217bc0de260eafe65adf33c7c7f4efb4a/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461573    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.949664222s: [/var/lib/containers/storage/overlay/a8644535bfb2d615f82e12d3561cc67e6a0f9b4f39ad5aba896c5f0fb4112025/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461598    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.949685717s: [/var/lib/containers/storage/overlay/dbfd981b1caa73c1407b2b269188d68007b420328aa49bd8049a813f4d3717d4/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461776    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.811947019s: [/var/lib/containers/storage/overlay/47152d6705729ca734188a2caea00e5357e35315d8778b07cc81c4b7a9e73973/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461802    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.769133443s: [/var/lib/containers/storage/overlay/e367a44158d4979ac308c95ff5fba6e221dcec76a57e3c0eb18b904cee3dafc9/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461825    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.703368273s: [/var/lib/containers/storage/overlay/129093fd31147dcb4550dea87313c5de6437232023e29ce2117872a4eb499954/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461848    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.322183148s: [/var/lib/containers/storage/overlay/a9ad6b8b34dedf157e0970b4c706b093111aaa8c561d5d57896cf48866bdd216/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461871    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 12.910557616s: [/var/lib/containers/storage/overlay/2f851245057b82bfcbb4de68e23434410ade9386602bb7ed619eba0c9daf4f73/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461894    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 12.90746862s: [/var/lib/containers/storage/overlay/80f3922871aeb52a7dd648e5a21a71892e73e9aa86b4f28be8d510aa7531a2cc/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.461918    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 12.907486383s: [/var/lib/containers/storage/overlay/15299c048d6d99ee9ba8eca0c42d32d2bc3d9bdf8c46efd01a77ea4f01bca381/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462684    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.582070509s: [/var/lib/containers/storage/overlay/5a1d520a71bd5be6625c336301a4a5c22f88239c319047e90cc898b85c1ada87/diff /var/log/pods/kube-system_konnectivity-agent-vwk66_324d72a4-a641-4b09-99d6-d682b5e2c958/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462699    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.571045385s: [/var/lib/containers/storage/overlay/0ec0e31cabfb28a7b957e19100eb32434c363d82921ae1183814cc87bc53d2c9/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pzj9l_801f804c-187a-4232-af07-412a115a2fef/northd/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462712    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.542030494s: [/var/lib/containers/storage/overlay/0f40ab6f09d416cbe3854a3898fcfceb99184a1a7c3ef458d91f2187cb315717/diff /var/log/pods/openshift-monitoring_thanos-querier-f96f5cd85-t7ndb_b96d72b2-bc45-4962-b529-f08155ad0215/kube-rbac-proxy-rules/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462722    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.542032934s: [/var/lib/containers/storage/overlay/7c77fb79af94a6539e27347b8d63f6b557a4fe985f9051608432f94184633ab9/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-js76b_8c893b43-cd05-45a0-9f7b-a2b5d72809c7/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462729    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.542035448s: [/var/lib/containers/storage/overlay/6d42a28cf861b642ba4bc06f3c1a0130d28d681e18e600fb2af5a044adf48411/diff /var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-136-38.ec2.internal_39a608f7d1c53c952b07789e6ea659e9/kube-rbac-proxy-crio/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462744    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.542016671s: [/var/lib/containers/storage/overlay/9897fa798adf0b887467e752d3a243d5232dfac2950acc532233b91831ad2955/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-xw2d4_066041b1-7eba-4371-9470-101a7bf8fcfc/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462746    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.542050001s: [/var/lib/containers/storage/overlay/d680aecd996fcaed785a8a61b443a788d0ee1b4e51893ea9bd226407d50d036a/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pzj9l_801f804c-187a-4232-af07-412a115a2fef/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462756    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.54202778s: [/var/lib/containers/storage/overlay/07c7ec972c6acc53d9c9b8b3d38d6c866eb67c5b771ca309c0bd41f4a4d282aa/diff /var/log/pods/openshift-monitoring_thanos-querier-f96f5cd85-t7ndb_b96d72b2-bc45-4962-b529-f08155ad0215/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462765    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.5820713s: [/var/lib/containers/storage/overlay/8fc2c6e6e2e78ead33ca917b72958601d1777888755f74a69e51da2e44857dd3/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pzj9l_801f804c-187a-4232-af07-412a115a2fef/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462770    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.504681766s: [/var/lib/containers/storage/overlay/99601e6ada822030b941736fa3d2e07cbd8592c04fc69abfd18ee6685a06de02/diff /var/log/pods/openshift-monitoring_telemeter-client-54969c98b4-7zdlx_dab249e6-5406-4ca6-87d5-d6fe41de7754/reload/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462783    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.460990517s: [/var/lib/containers/storage/overlay/1c6283c17fd5d689fc6496ff15ad7e5292807cf9138bd89ec7aaa995739871eb/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-js76b_8c893b43-cd05-45a0-9f7b-a2b5d72809c7/kube-rbac-proxy-self/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462785    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.494049095s: [/var/lib/containers/storage/overlay/b6f8f3c2556d89c1ff34ed1e7a52b81f84ceef8fc90ed8db7f3820f7c8eef970/diff /var/log/pods/openshift-monitoring_telemeter-client-54969c98b4-7zdlx_dab249e6-5406-4ca6-87d5-d6fe41de7754/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462796    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.460978942s: [/var/lib/containers/storage/overlay/eafbde367c60e934e00106f0d2a68149c6c2059c50a34e47b560716702e98915/diff /var/log/pods/openshift-dns_node-resolver-8jkmb_a09b130d-4249-4539-add0-e0aba69b752a/dns-node-resolver/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462802    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.460986116s: [/var/lib/containers/storage/overlay/175be1103f2ee140ddad7259bfe0325e068e062012b9da9f98b28455fc940815/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-xw2d4_066041b1-7eba-4371-9470-101a7bf8fcfc/kube-rbac-proxy-self/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462809    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.429783737s: [/var/lib/containers/storage/overlay/cce73e5b4f934be055d08520f7c7ff121159d6624738bd9846acfdbdabd32481/diff /var/log/pods/openshift-monitoring_node-exporter-wtbq7_3b64908a-fffb-4ec2-8697-51ea5db37f3b/node-exporter/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462818    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.428663026s: [/var/lib/containers/storage/overlay/8f7d81e4dc3411674d1c71d1eda7ce9acd95af7017e48611d3bef87eabab574f/diff /var/log/pods/openshift-monitoring_thanos-querier-f96f5cd85-t7ndb_b96d72b2-bc45-4962-b529-f08155ad0215/prom-label-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462824    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.428667614s: [/var/lib/containers/storage/overlay/67c1865571a924c15e9c00e50ca1d3e8f9ca02558858305c505505cdbb321df1/diff /var/log/pods/openshift-network-diagnostics_network-check-target-n8m99_0fadb982-a597-4489-ae16-3dc4fb5a7810/network-check-target-container/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462832    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.42868418s: [/var/lib/containers/storage/overlay/7a718033f8d571596f62556a2c3b40252cc913ab4c85536e79dff3703775d647/diff /var/log/pods/openshift-multus_multus-n4nsq_c2e1c568-867f-472f-9c39-47588d5f3fca/kube-multus/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462838    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.39380278s: [/var/lib/containers/storage/overlay/823156e2ea0b5f9042f9892c60ce5497a463ee31d98d266864e976ec42159514/diff /var/log/pods/openshift-network-operator_iptables-alerter-l7z5g_3017c536-859c-4b7b-b59e-f1d5ee77e117/iptables-alerter/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462849    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.393774165s: [/var/lib/containers/storage/overlay/494c5af9b22d1481e40ea0092a58718b8636a935decce497fcf3f9efa82c1b0e/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-js76b_8c893b43-cd05-45a0-9f7b-a2b5d72809c7/openshift-state-metrics/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462853    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.392701287s: [/var/lib/containers/storage/overlay/d9696e3a1398cdca3cdb4cde03f35eec881688dbf4f8dcbed5fb465d450fd4c4/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pzj9l_801f804c-187a-4232-af07-412a115a2fef/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462863    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.384186263s: [/var/lib/containers/storage/overlay/7daa39a208bc852418a43e0141735a918641cfc1d11735f97583efc97ca1bf10/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-xw2d4_066041b1-7eba-4371-9470-101a7bf8fcfc/kube-state-metrics/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462870    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.390581664s: [/var/lib/containers/storage/overlay/52816513ccf7d01733ec1d18a4bbf1e36242fa952a32716abf105905b41976c1/diff /var/log/pods/openshift-monitoring_telemeter-client-54969c98b4-7zdlx_dab249e6-5406-4ca6-87d5-d6fe41de7754/telemeter-client/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462883    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.384204922s: [/var/lib/containers/storage/overlay/befd3a8894a545d22b96a4ebd16d677697aa4d2490b4c8ea1adc610369313fe6/diff /var/log/pods/openshift-multus_network-metrics-daemon-4zxrn_ef2a700a-5061-402a-ab9c-29a44204054c/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462897    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.384215818s: [/var/lib/containers/storage/overlay/f62a07ce86e5ab6ebc55718b8215808b8ad2bf870da05d7d05c8ee39fc1033e8/diff /var/log/pods/openshift-monitoring_thanos-querier-f96f5cd85-t7ndb_b96d72b2-bc45-4962-b529-f08155ad0215/kube-rbac-proxy-metrics/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462915    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.384230761s: [/var/lib/containers/storage/overlay/df2e224c0ed7044efddcc05a2c6fc0af61d3e3025fd05e25a364685949febddf/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pzj9l_801f804c-187a-4232-af07-412a115a2fef/sbdb/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462924    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.388481349s: [/var/lib/containers/storage/overlay/adf7218733f1e5fa6dbb8c0e91b96f6f560926115a8b75da993b02550a4446d2/diff /var/log/pods/openshift-monitoring_node-exporter-wtbq7_3b64908a-fffb-4ec2-8697-51ea5db37f3b/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462931    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.383175933s: [/var/lib/containers/storage/overlay/1684c95a730744aa16084c3edd8950786a861a569f1cf805f2f26bec86304baf/diff /var/log/pods/openshift-dns_dns-default-245p6_3accc688-8a04-4462-9593-228a18548478/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462945    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.38318755s: [/var/lib/containers/storage/overlay/2347a383d05579172c7f5915bd7360ca8b282485e3ca52d785d4190198687b91/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pzj9l_801f804c-187a-4232-af07-412a115a2fef/nbdb/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462949    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.383168237s: [/var/lib/containers/storage/overlay/4b3d0e9ee1788f1aba6e4ba96d0125d9ef0e3b1b8240df9085bafa17932fc044/diff /var/log/pods/openshift-multus_multus-additional-cni-plugins-gd2nd_5183b0be-c47a-4e35-a8cb-77991e6f7208/kube-multus-additional-cni-plugins/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462958    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.351258511s: [/var/lib/containers/storage/overlay/7989e60c582f572f461e8685a0761dcc0041128cf10c2c44de4d175da79e26b8/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pzj9l_801f804c-187a-4232-af07-412a115a2fef/ovn-controller/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462971    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.351267644s: [/var/lib/containers/storage/overlay/5e039f25d4b69da4d528d012c81c4ee080eee41055c717dd434baee26e7f6f2f/diff /var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-wc4l2_4c2dd590-c639-42fe-8a02-4945b6403b3b/monitoring-plugin/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462971    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.359457818s: [/var/lib/containers/storage/overlay/76c44e526ac832272d3f739b22aac71b91f82da99551d123d7f740575c232923/diff /var/log/pods/openshift-insights_insights-runtime-extractor-2867l_79bf4395-e3d6-4071-aedd-548c185fa39d/extractor/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462985    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.351275844s: [/var/lib/containers/storage/overlay/3eef5983e5b58a66891288d39a7b06018b7737ce0268aae78109907d61a7a924/diff /var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-38.ec2.internal_39cf47f8a2a071911c6009d122ac3420/haproxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462989    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.351277912s: [/var/lib/containers/storage/overlay/20dc850cfa015e1e52318482efa9c27096e0bc36d965af9b2708698e9b4e4291/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pzj9l_801f804c-187a-4232-af07-412a115a2fef/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.462999    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.206852557s: [/var/lib/containers/storage/overlay/e6160a6ec3d8afb54c523726eee9707bec2bc7cebc581bf2405b6c71b9fb5b14/diff /var/log/pods/openshift-ingress-canary_ingress-canary-zv5gd_cd3c4499-8959-4616-842b-c556e7d23137/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.463003    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.206850495s: [/var/lib/containers/storage/overlay/499904119d88e1ef3520f7c4ece2fb323b27923649d373d8a665d20dfa7b2a60/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5787g_f683dbce-ebac-4d2c-ba04-aba3ac20792b/csi-node-driver-registrar/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.463013    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.205790811s: [/var/lib/containers/storage/overlay/4bdf6be06879d4fef0d5f815e38284f956c29f0af8db3b7c646f790d8b174abe/diff /var/log/pods/openshift-monitoring_thanos-querier-f96f5cd85-t7ndb_b96d72b2-bc45-4962-b529-f08155ad0215/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.463017    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.206863963s: [/var/lib/containers/storage/overlay/d19e4e170bf465c6d1ecd38d93de40132f50594784e0376c1915e38a17d5e23e/diff /var/log/pods/openshift-monitoring_metrics-server-f7fc46b4-vvxlv_baaab207-5534-4ef9-ac3a-9466ea5b900e/metrics-server/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.463030    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.813269859s: [/var/lib/containers/storage/overlay/41de065cfc10e06e76f753ea079dd91fe532cfcdb36e75826dd986006a925b7a/diff /var/log/pods/openshift-image-registry_node-ca-dmgg8_1f41e102-fab5-4206-9502-fbf09515069a/node-ca/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.463025    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.951115633s: [/var/lib/containers/storage/overlay/76096da963e64da254f678604b68229295fda7f1b50e96f0db56fa24e36ea18e/diff /var/log/pods/openshift-monitoring_thanos-querier-f96f5cd85-t7ndb_b96d72b2-bc45-4962-b529-f08155ad0215/thanos-query/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.463042    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.81325035s: [/var/lib/containers/storage/overlay/ea7cb5e4d409e7faa0f665ba4c5171a0f9a5c815aef8cb594a9425c4cbde4fdf/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5787g_f683dbce-ebac-4d2c-ba04-aba3ac20792b/csi-driver/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.463048    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.37298382s: [/var/lib/containers/storage/overlay/95021bc34770386f5c5ed7816ce82e7ab2c3c43fd7d91a1480e76205d445b18b/diff /var/log/pods/openshift-ingress_e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt_3969f40e-b7c2-4839-85ef-998ab9d08122/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.463056    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 12.508547028s: [/var/lib/containers/storage/overlay/5c587fa7f5e28bc3ee79c35c4b57b36368d332e4caef315b6543a0913edc5bb3/diff /var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-swf6j_1aadf369-2748-40a7-9990-6277f6aa995f/cert-manager-cainjector/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.463066    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.097318864s: [/var/lib/containers/storage/overlay/ee85a5c5c4941b4de3ecba32d3fda245ec13c7417867dc6b446884b63f3639db/diff /var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6b575997cc-zthg8_b8178c0e-7643-499f-9dc0-e04329809348/manager/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.463073    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 12.842197658s: [/var/lib/containers/storage/overlay/09f40f57f53a92d48b110ed204f57e0f124c6f10097439cf0437c45f2cab035a/diff /var/log/pods/openshift-ingress_e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj_2a7fa2cd-1234-4b80-b869-3ab7019d1cea/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.463080    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.087383046s: [/var/lib/containers/storage/overlay/5ccc2e5f347ec3bf66ddf008b2287ad57793cbde94e96cc5fb54bcf3b2e56044/diff /var/log/pods/openshift-ingress_e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4_65dfc02a-c81d-4366-a1a5-a5808d335aaf/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.463620    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.582673441s: [/var/lib/containers/storage/overlay/ba95b1b2c082fb93f2695c5e9c43eb049653e332151bbac4efa8fc7c5d7e3ede/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.463883    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.582873645s: [/var/lib/containers/storage/overlay/21fe740bb49d30b935589e086aee773306f1f13312433cb06d7180fd1bc45b23/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.465339    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.584199723s: [/var/lib/containers/storage/overlay/0c804bf029ccdb0c53d2786145f01158973ae81ed4910edbca1cc8eeef8b6ea9/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.465488    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.584349811s: [/var/lib/containers/storage/overlay/a6cb7c469d2bdde759c7a0cbe39ecac6a87002b2cf0e14ed56be05abb4bb9eb2/diff /var/log/pods/openshift-dns_dns-default-245p6_3accc688-8a04-4462-9593-228a18548478/dns/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.465708    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.584485263s: [/var/lib/containers/storage/overlay/c5cf8b9f7a36208e47beae605a78e8085e48fe792ddad1433c4f08adef2ba015/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-frxl2_bbe53da9-f387-477c-8591-f5c37f42eaf5/tuned/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:23.828085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.465742    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.584515985s: [/var/lib/containers/storage/overlay/caf21382ce4bfe417c814eada36eab8e1a8181a8ab3cf727adde9bad89b9922e/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.465949    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.584679487s: [/var/lib/containers/storage/overlay/8ca7d8c26b05fe15d5edfbfd4ff809607f67b6fa42d4906b231021389d141f7b/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.471483    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.579894139s: [/var/lib/containers/storage/overlay/13ee83133a5fa86a829c4f9c48bf3fe390c624b4a12d0c10ab5a4df98dc3aac5/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.471726    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.580116566s: [/var/lib/containers/storage/overlay/f63c250f662464c3e3e811469e63052a9804e12393ac0bbe1f23e03ac6d42252/diff /var/log/pods/kube-system_global-pull-secret-syncer-vth7d_ae94d17c-b881-46e1-8063-e88c9dc53369/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.471764    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.580152689s: [/var/lib/containers/storage/overlay/0becafeebe30c865a220f9d37e3dc1536fdd9e4563a250e69b21b75346818345/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.471799    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.580184791s: [/var/lib/containers/storage/overlay/51d6a1167f8955b4036d5ff24be2f725ca11572ee163d856f637dfaf065ed6ab/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.471924    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.580306311s: [/var/lib/containers/storage/overlay/862f2b2cba49f9fb142d99c2a13d3c3803204d6bb3c1df6eb5e6754df7205d99/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5787g_f683dbce-ebac-4d2c-ba04-aba3ac20792b/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.472038    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.580416786s: [/var/lib/containers/storage/overlay/f1de0dd5845a0249eb57540359fb3073525856975558cdc40bb12b3e127ec204/diff /var/log/pods/openshift-multus_network-metrics-daemon-4zxrn_ef2a700a-5061-402a-ab9c-29a44204054c/network-metrics-daemon/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.472122    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.580474807s: [/var/lib/containers/storage/overlay/8e193bf1e3033d491daf91dc88813d6bbda392775536624062ee44757d03d7d0/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.472198    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.580545446s: [/var/lib/containers/storage/overlay/2a8bb9b23edea87881d86fa86aee98440bb5e931dd4c6ffbcf92569bce8a1b03/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.472226    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.58058468s: [/var/lib/containers/storage/overlay/2cc7a42713ed864368c9745b32c5cfead5f02b67542359ed690c679738d6b6b7/diff /var/log/pods/openshift-insights_insights-runtime-extractor-2867l_79bf4395-e3d6-4071-aedd-548c185fa39d/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.472238    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.580586786s: [/var/lib/containers/storage/overlay/4adcf16154d8e81a744147279c4f2f0e582a599cf8c9df660abe4a628d627814/diff ]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.472273    7082 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.58062542s: [/var/lib/containers/storage/overlay/af6b4b07d27749e3d5723cf62d0bad5c410317030f5c96b92645692441636fe7/diff /var/log/pods/openshift-insights_insights-runtime-extractor-2867l_79bf4395-e3d6-4071-aedd-548c185fa39d/exporter/0.log]; will not log again for this container unless duration exceeds 2s
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.487659    7082 patch_prober.go:29] interesting pod/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.487692    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2" podUID="43619c62-7adb-49b0-94c8-870ec330cd19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.903662    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.903737    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:22.907490    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q" podUID="86f63896-629a-4327-a138-279748c035bd" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.60:50052\" within 1s: context deadline exceeded"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:08:23.160341    7082 logging.go:55] [core] [Channel #1815 SubChannel #1817]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.60:50052", ServerName: "10.134.0.60:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:54212->10.134.0.60:50052: use of closed network connection"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:23.188973    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb" podUID="7b205272-e4c0-432c-860d-162d9e80f950" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.61:50052\" within 1s: context deadline exceeded"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:08:23.189370    7082 logging.go:55] [core] [Channel #1814 SubChannel #1819]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.61:50052", ServerName: "10.134.0.61:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:39360->10.134.0.61:50052: use of closed network connection"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:23.194540    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk" podUID="e07e2d2d-0845-4243-82ac-6a17427f2290" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.156:50052\" within 1s: context deadline exceeded"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:08:23.194971    7082 logging.go:55] [core] [Channel #1816 SubChannel #1818]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.156:50052", ServerName: "10.134.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:54236->10.134.0.156:50052: use of closed network connection"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:23.204837    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" podUID="4684c6fd-8ac7-4005-a20d-48396c0ab981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:23.332488    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr" podUID="e3a92df7-921d-4f5f-a18b-0c9494dfba37" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.74:50052\" within 1s: context deadline exceeded"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:08:23.336321    7082 logging.go:55] [core] [Channel #1821 SubChannel #1823]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.74:50052", ServerName: "10.134.0.74:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:34334->10.134.0.74:50052: use of closed network connection"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:23.380724    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx" podUID="0369fe15-bc84-442e-973c-fde55e95785a" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:23.397776    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:23.397847    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8" podUID="423fc4e9-6575-4690-9263-83ffff40dfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:23.438761    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2" podUID="efe88fdd-8cf4-4758-8d81-546d50c2786f" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:23.439728    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8" podUID="b8178c0e-7643-499f-9dc0-e04329809348" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:23.442729    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8" podUID="b8178c0e-7643-499f-9dc0-e04329809348" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:23.498741    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x" podUID="79e8c570-3e0f-40b7-9696-81b84b10e8eb" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:23.502801    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" podUID="4684c6fd-8ac7-4005-a20d-48396c0ab981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:23.522301    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk" podUID="b742a630-e75b-4040-8521-12388f3d0427" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:24.242795    7082 trace.go:236] Trace[388763647]: "iptables ChainExists" (29-Oct-2026 22:08:07.881) (total time: 16361ms):
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: Trace[388763647]: [16.361304454s] [16.361304454s] END
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:24.252971    7082 trace.go:236] Trace[369188271]: "iptables ChainExists" (29-Oct-2026 22:08:07.880) (total time: 16371ms):
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: Trace[369188271]: [16.371995939s] [16.371995939s] END
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:08:24.444356    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b42b80_df94_4d12_80cb_5814f733d918.slice/crio-conmon-1b1ce3527cb87d2cb6197f058b48c9bd38090af9b9c969e48a2778ec15377f45.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:08:24.460149    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b42b80_df94_4d12_80cb_5814f733d918.slice/crio-conmon-1b1ce3527cb87d2cb6197f058b48c9bd38090af9b9c969e48a2778ec15377f45.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:08:24.477936    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b42b80_df94_4d12_80cb_5814f733d918.slice/crio-conmon-1b1ce3527cb87d2cb6197f058b48c9bd38090af9b9c969e48a2778ec15377f45.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:24.482738    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8" podUID="b8178c0e-7643-499f-9dc0-e04329809348" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:24.586014    7082 scope.go:122] "RemoveContainer" containerID="1b59e04effa4a26eb82a74fee0c0cbcc8564bf054116bbfa2f47c2bbb5488d58"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:24.629170    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2198e677-436e-4fea-838e-cbc5881dc6e8" path="/var/lib/kubelet/pods/2198e677-436e-4fea-838e-cbc5881dc6e8/volumes"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:08:24.855071    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.364s"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:24.855102    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"]
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:24.855183    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:24.855209    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dfe75a0-n4l5n"]
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:24.857010    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:24.962666    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b937919a-4a86-4a6a-be3e-a0c9a105d0f7" path="/var/lib/kubelet/pods/b937919a-4a86-4a6a-be3e-a0c9a105d0f7/volumes"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.012781    7082 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-5787g container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.012834    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g" podUID="f683dbce-ebac-4d2c-ba04-aba3ac20792b" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.115783    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.115846    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.115866    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"]
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.115893    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a248d5-lhmll"]
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.116067    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.116136    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.116155    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.116315    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.116460    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.117410    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.117450    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.117466    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.117513    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.117557    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.117596    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.543751    7082 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-5787g container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.543825    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g" podUID="f683dbce-ebac-4d2c-ba04-aba3ac20792b" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.725990    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a87ce0-f034-4a27-aee7-2045c8c21ab4" path="/var/lib/kubelet/pods/86a87ce0-f034-4a27-aee7-2045c8c21ab4/volumes"
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.759617    7082 generic.go:356] "Generic (PLEG): container finished" podID="b8178c0e-7643-499f-9dc0-e04329809348" containerID="adeeace6427d775b3c7541202cba74c1947e3885f9d34f2149cd465885c6d6b7" exitCode=0
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.759765    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8" event={"ID":"b8178c0e-7643-499f-9dc0-e04329809348","Type":"ContainerDied","Data":"adeeace6427d775b3c7541202cba74c1947e3885f9d34f2149cd465885c6d6b7"}
Oct 29 22:08:25.810918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.760383    7082 scope.go:122] "RemoveContainer" containerID="adeeace6427d775b3c7541202cba74c1947e3885f9d34f2149cd465885c6d6b7"
Oct 29 22:08:26.233904 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.875441    7082 generic.go:356] "Generic (PLEG): container finished" podID="f4b42b80-df94-4d12-80cb-5814f733d918" containerID="1b1ce3527cb87d2cb6197f058b48c9bd38090af9b9c969e48a2778ec15377f45" exitCode=1
Oct 29 22:08:26.233904 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.875563    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs" event={"ID":"f4b42b80-df94-4d12-80cb-5814f733d918","Type":"ContainerDied","Data":"1b1ce3527cb87d2cb6197f058b48c9bd38090af9b9c969e48a2778ec15377f45"}
Oct 29 22:08:26.233904 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.914130    7082 generic.go:356] "Generic (PLEG): container finished" podID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerID="d82bec39cf4f34f2d7688641833b8b0044ac07fd0cfe0d375d516046a66ddf87" exitCode=0
Oct 29 22:08:26.233904 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.914263    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" event={"ID":"76a92f63-b5c0-4f00-9976-5bbc789b2bed","Type":"ContainerDied","Data":"d82bec39cf4f34f2d7688641833b8b0044ac07fd0cfe0d375d516046a66ddf87"}
Oct 29 22:08:26.233904 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.914992    7082 scope.go:122] "RemoveContainer" containerID="1b1ce3527cb87d2cb6197f058b48c9bd38090af9b9c969e48a2778ec15377f45"
Oct 29 22:08:26.233904 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:25.952469    7082 scope.go:122] "RemoveContainer" containerID="0d6511bba247899bf06112c391bf506fe850700a1242bdbc63fd679647b4cd67"
Oct 29 22:08:26.355823 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.310296    7082 generic.go:356] "Generic (PLEG): container finished" podID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerID="efee2de6f820641c7f87224c2c71d0a7984816e3e82ddfcec7b2ccc1c2b84800" exitCode=0
Oct 29 22:08:26.355823 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.310518    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" event={"ID":"4d1d3eec-cb23-48d7-bc11-bbe55580eee2","Type":"ContainerDied","Data":"efee2de6f820641c7f87224c2c71d0a7984816e3e82ddfcec7b2ccc1c2b84800"}
Oct 29 22:08:26.355823 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.310590    7082 scope.go:122] "RemoveContainer" containerID="b2562dad0650731c90a8b6a67a45a6e3a7b099cda3ab3b46cef1356296af816c"
Oct 29 22:08:26.355823 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.349827    7082 generic.go:356] "Generic (PLEG): container finished" podID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" containerID="800159fb2e9afbc67db3c9c6b50fec50328770c8edbf2efd22113a417c37e2df" exitCode=0
Oct 29 22:08:26.355823 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.350226    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" event={"ID":"6b822e3a-e780-4b65-98bc-f0c26990bfcf","Type":"ContainerDied","Data":"800159fb2e9afbc67db3c9c6b50fec50328770c8edbf2efd22113a417c37e2df"}
Oct 29 22:08:26.442096 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.403955    7082 scope.go:122] "RemoveContainer" containerID="b4e67939aad355f6758ce00510b08f1f9267adfae65ce7bdb7d6f59abf660b11"
Oct 29 22:08:26.513598 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.492642    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"
Oct 29 22:08:26.667713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.667677    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-xks-ca\") pod \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\" (UID: \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\") "
Oct 29 22:08:26.667891 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.667812    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-openshift-ca\") pod \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\" (UID: \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\") "
Oct 29 22:08:26.667974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.667958    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-kube-api-access-lnvbm\" (UniqueName: \"kubernetes.io/projected/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-kube-api-access-lnvbm\") pod \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\" (UID: \"9ce8bea9-8a70-4e28-9d56-53a0932d1b93\") "
Oct 29 22:08:26.696267 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.696165    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-openshift-ca" pod "9ce8bea9-8a70-4e28-9d56-53a0932d1b93" (UID: "9ce8bea9-8a70-4e28-9d56-53a0932d1b93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:26.703645 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.703593    7082 patch_prober.go:29] interesting pod/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:08:26.703786 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.703674    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2" podUID="43619c62-7adb-49b0-94c8-870ec330cd19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:26.770267 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.770238    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:26.835880 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:26.835706    7082 scope.go:122] "RemoveContainer" containerID="af5fd0b76632ae6e91dc828e1dd073d995a0a25af855b409c93e3a57c390c6ca"
Oct 29 22:08:27.367016 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:27.366990    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"
Oct 29 22:08:27.367445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:27.367023    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n" event={"ID":"9ce8bea9-8a70-4e28-9d56-53a0932d1b93","Type":"ContainerDied","Data":"f54bc0408221e600a03c63d05bd17d1c001aec126a76f66c1726ab39a427731c"}
Oct 29 22:08:27.367445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:27.367078    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f54bc0408221e600a03c63d05bd17d1c001aec126a76f66c1726ab39a427731c"
Oct 29 22:08:27.476756 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:27.476729    7082 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 29 22:08:27.476878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:27.476777    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 29 22:08:27.524282 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:27.524232    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-xks-ca" pod "9ce8bea9-8a70-4e28-9d56-53a0932d1b93" (UID: "9ce8bea9-8a70-4e28-9d56-53a0932d1b93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:08:27.524282 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:27.524283    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-kube-api-access-lnvbm" pod "9ce8bea9-8a70-4e28-9d56-53a0932d1b93" (UID: "9ce8bea9-8a70-4e28-9d56-53a0932d1b93"). InnerVolumeSpecName "kube-api-access-lnvbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:27.560908 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:27.560447    7082 scope.go:122] "RemoveContainer" containerID="f7e25d5bc307084fb76916e2f8002ae2805164f3a4a99b5cddeac79f39ba28a5"
Oct 29 22:08:27.586155 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:27.586094    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:27.586155 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:27.586131    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnvbm\" (UniqueName: \"kubernetes.io/projected/9ce8bea9-8a70-4e28-9d56-53a0932d1b93-kube-api-access-lnvbm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:27.636532 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:27.629504    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 22:08:28.082187 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.051022    7082 scope.go:122] "RemoveContainer" containerID="8e3b4ead1cb25ff086abe1b94e81d1bed1301c681d943b6a18f637d6911157a3"
Oct 29 22:08:28.167116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.167088    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:08:28.217806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.199240    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:08:28.217806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.215843    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:08:28.311335 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.311310    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-credential-socket\") pod \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") "
Oct 29 22:08:28.311449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.311400    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istiod-ca-cert\") pod \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") "
Oct 29 22:08:28.311449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.311438    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-certs\") pod \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") "
Oct 29 22:08:28.311600 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.311508    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-credential-socket\") pod \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") "
Oct 29 22:08:28.311600 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.311569    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istiod-ca-cert\") pod \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") "
Oct 29 22:08:28.311924 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.311644    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-data\") pod \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") "
Oct 29 22:08:28.311924 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.311724    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-kube-api-access-dvzqn\" (UniqueName: \"kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-kube-api-access-dvzqn\") pod \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") "
Oct 29 22:08:28.311924 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.311772    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-data\") pod \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") "
Oct 29 22:08:28.311924 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.311771    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-credential-socket" pod "6b822e3a-e780-4b65-98bc-f0c26990bfcf" (UID: "6b822e3a-e780-4b65-98bc-f0c26990bfcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.311924 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.311809    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-podinfo\") pod \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") "
Oct 29 22:08:28.313208 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.313186    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-envoy\") pod \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") "
Oct 29 22:08:28.313475 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.313458    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-kube-api-access-rk6jc\" (UniqueName: \"kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-kube-api-access-rk6jc\") pod \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") "
Oct 29 22:08:28.313646 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.313614    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-credential-socket\") pod \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") "
Oct 29 22:08:28.313798 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.313783    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-certs\") pod \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") "
Oct 29 22:08:28.313933 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.313919    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-token\" (UniqueName: \"kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-token\") pod \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") "
Oct 29 22:08:28.314151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.314135    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-data\") pod \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") "
Oct 29 22:08:28.314295 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.314278    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istiod-ca-cert\") pod \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") "
Oct 29 22:08:28.315843 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.315822    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-socket\") pod \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") "
Oct 29 22:08:28.316016 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.316000    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-kube-api-access-lfrv8\" (UniqueName: \"kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-kube-api-access-lfrv8\") pod \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") "
Oct 29 22:08:28.316161 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.316145    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-envoy\") pod \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") "
Oct 29 22:08:28.317101 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.317076    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-podinfo\") pod \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") "
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.317134    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-envoy\") pod \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") "
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.317207    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-podinfo\") pod \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") "
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.317246    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-socket\") pod \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") "
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.317275    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-token\" (UniqueName: \"kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-token\") pod \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") "
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.317320    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-certs\") pod \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\" (UID: \"4d1d3eec-cb23-48d7-bc11-bbe55580eee2\") "
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.317366    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-token\" (UniqueName: \"kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-token\") pod \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\" (UID: \"6b822e3a-e780-4b65-98bc-f0c26990bfcf\") "
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.317402    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-socket\") pod \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\" (UID: \"76a92f63-b5c0-4f00-9976-5bbc789b2bed\") "
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.318714    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.337266    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istiod-ca-cert" pod "76a92f63-b5c0-4f00-9976-5bbc789b2bed" (UID: "76a92f63-b5c0-4f00-9976-5bbc789b2bed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.338047    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istiod-ca-cert" pod "6b822e3a-e780-4b65-98bc-f0c26990bfcf" (UID: "6b822e3a-e780-4b65-98bc-f0c26990bfcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.338891    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-certs" pod "6b822e3a-e780-4b65-98bc-f0c26990bfcf" (UID: "6b822e3a-e780-4b65-98bc-f0c26990bfcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.338923    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-data" pod "6b822e3a-e780-4b65-98bc-f0c26990bfcf" (UID: "6b822e3a-e780-4b65-98bc-f0c26990bfcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.338969    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-socket" pod "6b822e3a-e780-4b65-98bc-f0c26990bfcf" (UID: "6b822e3a-e780-4b65-98bc-f0c26990bfcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.344471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.339541    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-credential-socket" pod "4d1d3eec-cb23-48d7-bc11-bbe55580eee2" (UID: "4d1d3eec-cb23-48d7-bc11-bbe55580eee2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.354752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.349873    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-kube-api-access-dvzqn" pod "4d1d3eec-cb23-48d7-bc11-bbe55580eee2" (UID: "4d1d3eec-cb23-48d7-bc11-bbe55580eee2"). InnerVolumeSpecName "kube-api-access-dvzqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:28.354752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.349870    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-kube-api-access-rk6jc" pod "6b822e3a-e780-4b65-98bc-f0c26990bfcf" (UID: "6b822e3a-e780-4b65-98bc-f0c26990bfcf"). InnerVolumeSpecName "kube-api-access-rk6jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:28.354752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.349900    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-podinfo" pod "6b822e3a-e780-4b65-98bc-f0c26990bfcf" (UID: "6b822e3a-e780-4b65-98bc-f0c26990bfcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:28.354752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.349979    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-envoy" pod "6b822e3a-e780-4b65-98bc-f0c26990bfcf" (UID: "6b822e3a-e780-4b65-98bc-f0c26990bfcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.356577 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.356546    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-certs" pod "76a92f63-b5c0-4f00-9976-5bbc789b2bed" (UID: "76a92f63-b5c0-4f00-9976-5bbc789b2bed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.356732 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.356680    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-credential-socket" pod "76a92f63-b5c0-4f00-9976-5bbc789b2bed" (UID: "76a92f63-b5c0-4f00-9976-5bbc789b2bed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.356823 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.356800    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-podinfo" pod "76a92f63-b5c0-4f00-9976-5bbc789b2bed" (UID: "76a92f63-b5c0-4f00-9976-5bbc789b2bed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:28.357141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.356890    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-token" pod "6b822e3a-e780-4b65-98bc-f0c26990bfcf" (UID: "6b822e3a-e780-4b65-98bc-f0c26990bfcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:28.381901 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.381873    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8" event={"ID":"b8178c0e-7643-499f-9dc0-e04329809348","Type":"ContainerStarted","Data":"2031e568e670cb6e890c6b572a627add49e9e220de420f953e26acc5fed23d96"}
Oct 29 22:08:28.383522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.383495    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr" event={"ID":"76a92f63-b5c0-4f00-9976-5bbc789b2bed","Type":"ContainerDied","Data":"68a9d28d2863472363e495dcabe6b638ff5cad50283f9ded1c9ce7e3815f2893"}
Oct 29 22:08:28.383522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.383514    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"
Oct 29 22:08:28.383778 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.383558    7082 scope.go:122] "RemoveContainer" containerID="d82bec39cf4f34f2d7688641833b8b0044ac07fd0cfe0d375d516046a66ddf87"
Oct 29 22:08:28.385506 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.385476    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs" event={"ID":"4d1d3eec-cb23-48d7-bc11-bbe55580eee2","Type":"ContainerDied","Data":"fb88f5c6b39d9f34c16efecb401e32686b1b1a4f84ecb9a2e102380bd8065180"}
Oct 29 22:08:28.385614 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.385598    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"
Oct 29 22:08:28.387432 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.387407    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq" event={"ID":"6b822e3a-e780-4b65-98bc-f0c26990bfcf","Type":"ContainerDied","Data":"ee2b5be7120a269def776611ce4ebf0d6c618f329c83b31c4732b8d0c3663ed0"}
Oct 29 22:08:28.387535 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.387502    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.397751    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-token" pod "76a92f63-b5c0-4f00-9976-5bbc789b2bed" (UID: "76a92f63-b5c0-4f00-9976-5bbc789b2bed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.397897    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-kube-api-access-lfrv8" pod "76a92f63-b5c0-4f00-9976-5bbc789b2bed" (UID: "76a92f63-b5c0-4f00-9976-5bbc789b2bed"). InnerVolumeSpecName "kube-api-access-lfrv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.397913    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-envoy" pod "76a92f63-b5c0-4f00-9976-5bbc789b2bed" (UID: "76a92f63-b5c0-4f00-9976-5bbc789b2bed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.397953    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-data" pod "76a92f63-b5c0-4f00-9976-5bbc789b2bed" (UID: "76a92f63-b5c0-4f00-9976-5bbc789b2bed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.398971    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-socket" pod "76a92f63-b5c0-4f00-9976-5bbc789b2bed" (UID: "76a92f63-b5c0-4f00-9976-5bbc789b2bed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.399164    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-socket" pod "4d1d3eec-cb23-48d7-bc11-bbe55580eee2" (UID: "4d1d3eec-cb23-48d7-bc11-bbe55580eee2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.399280    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istiod-ca-cert" pod "4d1d3eec-cb23-48d7-bc11-bbe55580eee2" (UID: "4d1d3eec-cb23-48d7-bc11-bbe55580eee2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.399299    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-data" pod "4d1d3eec-cb23-48d7-bc11-bbe55580eee2" (UID: "4d1d3eec-cb23-48d7-bc11-bbe55580eee2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.399501    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-certs" pod "4d1d3eec-cb23-48d7-bc11-bbe55580eee2" (UID: "4d1d3eec-cb23-48d7-bc11-bbe55580eee2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.403723    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-token" pod "4d1d3eec-cb23-48d7-bc11-bbe55580eee2" (UID: "4d1d3eec-cb23-48d7-bc11-bbe55580eee2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420492    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420516    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420528    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420539    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420550    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420560    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420570    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420585    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420597    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420610    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420643    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420655    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvzqn\" (UniqueName: \"kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-kube-api-access-dvzqn\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420668    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420680    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420692    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk6jc\" (UniqueName: \"kubernetes.io/projected/6b822e3a-e780-4b65-98bc-f0c26990bfcf-kube-api-access-rk6jc\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420705    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420717    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b822e3a-e780-4b65-98bc-f0c26990bfcf-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420729    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420741    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420752    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420763    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420775    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfrv8\" (UniqueName: \"kubernetes.io/projected/76a92f63-b5c0-4f00-9976-5bbc789b2bed-kube-api-access-lfrv8\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420788    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.434460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.420801    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a92f63-b5c0-4f00-9976-5bbc789b2bed-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.516543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.443418    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-podinfo" pod "4d1d3eec-cb23-48d7-bc11-bbe55580eee2" (UID: "4d1d3eec-cb23-48d7-bc11-bbe55580eee2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:28.516543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.443437    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-envoy" pod "4d1d3eec-cb23-48d7-bc11-bbe55580eee2" (UID: "4d1d3eec-cb23-48d7-bc11-bbe55580eee2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:28.516543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.483692    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"]
Oct 29 22:08:28.516543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.490641    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-2vdtq"]
Oct 29 22:08:28.533449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.522468    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.533449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.522488    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d1d3eec-cb23-48d7-bc11-bbe55580eee2-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.631245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.584753    7082 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-5787g container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:08:28.631245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.584822    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5787g" podUID="f683dbce-ebac-4d2c-ba04-aba3ac20792b" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:08:28.819660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:28.814448    7082 scope.go:122] "RemoveContainer" containerID="efee2de6f820641c7f87224c2c71d0a7984816e3e82ddfcec7b2ccc1c2b84800"
Oct 29 22:08:29.033664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:29.023881    7082 scope.go:122] "RemoveContainer" containerID="800159fb2e9afbc67db3c9c6b50fec50328770c8edbf2efd22113a417c37e2df"
Oct 29 22:08:29.033664 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:08:29.029580    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efee2de6f820641c7f87224c2c71d0a7984816e3e82ddfcec7b2ccc1c2b84800\": container with ID starting with efee2de6f820641c7f87224c2c71d0a7984816e3e82ddfcec7b2ccc1c2b84800 not found: ID does not exist" containerID="efee2de6f820641c7f87224c2c71d0a7984816e3e82ddfcec7b2ccc1c2b84800"
Oct 29 22:08:29.033664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:29.029654    7082 kuberuntime_gc.go:364] "Error getting ContainerStatus for containerID" containerID="efee2de6f820641c7f87224c2c71d0a7984816e3e82ddfcec7b2ccc1c2b84800" err="rpc error: code = NotFound desc = could not find container \"efee2de6f820641c7f87224c2c71d0a7984816e3e82ddfcec7b2ccc1c2b84800\": container with ID starting with efee2de6f820641c7f87224c2c71d0a7984816e3e82ddfcec7b2ccc1c2b84800 not found: ID does not exist"
Oct 29 22:08:29.033664 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:08:29.029691    7082 kuberuntime_gc.go:392] "Failed to remove container log dead symlink" err="remove /var/log/containers/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs_openshift-ingress_istio-proxy-efee2de6f820641c7f87224c2c71d0a7984816e3e82ddfcec7b2ccc1c2b84800.log: no such file or directory" path="/var/log/containers/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs_openshift-ingress_istio-proxy-efee2de6f820641c7f87224c2c71d0a7984816e3e82ddfcec7b2ccc1c2b84800.log"
Oct 29 22:08:29.065592 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:29.064801    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"]
Oct 29 22:08:29.077865 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:08:29.069870    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:08:29.272313 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:29.272235    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-4trvs"]
Oct 29 22:08:29.324706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:29.318812    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"]
Oct 29 22:08:29.418657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:29.403850    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-pntbr"]
Oct 29 22:08:29.512481 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:29.512448    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8"
Oct 29 22:08:29.531349 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:29.531291    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"]
Oct 29 22:08:29.818673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:29.818479    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" path="/var/lib/kubelet/pods/4d1d3eec-cb23-48d7-bc11-bbe55580eee2/volumes"
Oct 29 22:08:29.945304 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:29.944449    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" path="/var/lib/kubelet/pods/6b822e3a-e780-4b65-98bc-f0c26990bfcf/volumes"
Oct 29 22:08:30.007242 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:30.005011    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" path="/var/lib/kubelet/pods/76a92f63-b5c0-4f00-9976-5bbc789b2bed/volumes"
Oct 29 22:08:30.065675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:30.065647    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380eea-2zqq7"]
Oct 29 22:08:30.529444 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:30.528578    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs" event={"ID":"f4b42b80-df94-4d12-80cb-5814f733d918","Type":"ContainerStarted","Data":"a131863b0a7b1be97323aeace145cb449dc0ecaff471ceca5df18a39aa31e973"}
Oct 29 22:08:30.529444 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:30.528805    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 22:08:31.099547 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:31.099510    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"]
Oct 29 22:08:31.105351 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:31.105324    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-178f1555-t8brh"]
Oct 29 22:08:31.633989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:31.633962    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13290d1-0bf7-40dc-9e78-6e450afbda03" path="/var/lib/kubelet/pods/c13290d1-0bf7-40dc-9e78-6e450afbda03/volumes"
Oct 29 22:08:31.634678 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:31.634650    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f184d9db-6a8f-4e23-92f5-f36d4108ac1e" path="/var/lib/kubelet/pods/f184d9db-6a8f-4e23-92f5-f36d4108ac1e/volumes"
Oct 29 22:08:32.458868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.458836    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:08:32.491638 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.491598    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"]
Oct 29 22:08:32.492509 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492491    7082 cpu_manager.go:414] "containerMap: removing container" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy"
Oct 29 22:08:32.492597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492509    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy"
Oct 29 22:08:32.492597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492528    7082 cpu_manager.go:414] "containerMap: removing container" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy"
Oct 29 22:08:32.492597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492533    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy"
Oct 29 22:08:32.492597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492544    7082 cpu_manager.go:414] "containerMap: removing container" podUID="43619c62-7adb-49b0-94c8-870ec330cd19" containerName="istio-proxy"
Oct 29 22:08:32.492597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492549    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43619c62-7adb-49b0-94c8-870ec330cd19" containerName="istio-proxy"
Oct 29 22:08:32.492597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492586    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9ce8bea9-8a70-4e28-9d56-53a0932d1b93" containerName="revoke-keys"
Oct 29 22:08:32.492597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492591    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce8bea9-8a70-4e28-9d56-53a0932d1b93" containerName="revoke-keys"
Oct 29 22:08:32.492597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492600    7082 cpu_manager.go:414] "containerMap: removing container" podUID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" containerName="istio-proxy"
Oct 29 22:08:32.492928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492605    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" containerName="istio-proxy"
Oct 29 22:08:32.492928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492783    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce8bea9-8a70-4e28-9d56-53a0932d1b93" containerName="revoke-keys"
Oct 29 22:08:32.492928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492793    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b822e3a-e780-4b65-98bc-f0c26990bfcf" containerName="istio-proxy"
Oct 29 22:08:32.492928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492801    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy"
Oct 29 22:08:32.492928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492811    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy"
Oct 29 22:08:32.492928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492819    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a92f63-b5c0-4f00-9976-5bbc789b2bed" containerName="istio-proxy"
Oct 29 22:08:32.492928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.492825    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="43619c62-7adb-49b0-94c8-870ec330cd19" containerName="istio-proxy"
Oct 29 22:08:32.504922 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.504896    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.544786 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.544750    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-data\") pod \"43619c62-7adb-49b0-94c8-870ec330cd19\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") "
Oct 29 22:08:32.544951 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.544886    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-socket\") pod \"43619c62-7adb-49b0-94c8-870ec330cd19\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") "
Oct 29 22:08:32.545010 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.544980    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-credential-socket\") pod \"43619c62-7adb-49b0-94c8-870ec330cd19\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") "
Oct 29 22:08:32.545066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.545016    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43619c62-7adb-49b0-94c8-870ec330cd19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43619c62-7adb-49b0-94c8-870ec330cd19-istiod-ca-cert\") pod \"43619c62-7adb-49b0-94c8-870ec330cd19\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") "
Oct 29 22:08:32.545066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.545047    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43619c62-7adb-49b0-94c8-870ec330cd19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43619c62-7adb-49b0-94c8-870ec330cd19-istio-podinfo\") pod \"43619c62-7adb-49b0-94c8-870ec330cd19\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") "
Oct 29 22:08:32.546037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.545184    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-kube-api-access-qtkg5\" (UniqueName: \"kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-kube-api-access-qtkg5\") pod \"43619c62-7adb-49b0-94c8-870ec330cd19\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") "
Oct 29 22:08:32.546037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.545297    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-envoy\") pod \"43619c62-7adb-49b0-94c8-870ec330cd19\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") "
Oct 29 22:08:32.546037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.545401    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-certs\") pod \"43619c62-7adb-49b0-94c8-870ec330cd19\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") "
Oct 29 22:08:32.546037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.545459    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-istio-token\" (UniqueName: \"kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-istio-token\") pod \"43619c62-7adb-49b0-94c8-870ec330cd19\" (UID: \"43619c62-7adb-49b0-94c8-870ec330cd19\") "
Oct 29 22:08:32.546037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.545890    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa3614e-71e6-4d35-9046-8876a080c622-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.546037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.545928    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.546037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.546012    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.551789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.546085    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.551789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.546268    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.551789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.546435    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.551789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.546465    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77wbw\" (UniqueName: \"kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-kube-api-access-77wbw\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.551789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.546584    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.551789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.546669    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa3614e-71e6-4d35-9046-8876a080c622-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.551789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.548720    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-credential-socket" pod "43619c62-7adb-49b0-94c8-870ec330cd19" (UID: "43619c62-7adb-49b0-94c8-870ec330cd19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:32.552365 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.552109    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-certs" pod "43619c62-7adb-49b0-94c8-870ec330cd19" (UID: "43619c62-7adb-49b0-94c8-870ec330cd19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:32.552365 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.552198    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-data" pod "43619c62-7adb-49b0-94c8-870ec330cd19" (UID: "43619c62-7adb-49b0-94c8-870ec330cd19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:32.552674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.552408    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43619c62-7adb-49b0-94c8-870ec330cd19-istiod-ca-cert" pod "43619c62-7adb-49b0-94c8-870ec330cd19" (UID: "43619c62-7adb-49b0-94c8-870ec330cd19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:32.552674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.552494    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43619c62-7adb-49b0-94c8-870ec330cd19-istio-podinfo" pod "43619c62-7adb-49b0-94c8-870ec330cd19" (UID: "43619c62-7adb-49b0-94c8-870ec330cd19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:32.552674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.552515    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-kube-api-access-qtkg5" pod "43619c62-7adb-49b0-94c8-870ec330cd19" (UID: "43619c62-7adb-49b0-94c8-870ec330cd19"). InnerVolumeSpecName "kube-api-access-qtkg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:32.552674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.552562    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-envoy" pod "43619c62-7adb-49b0-94c8-870ec330cd19" (UID: "43619c62-7adb-49b0-94c8-870ec330cd19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:32.552923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.552673    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-socket" pod "43619c62-7adb-49b0-94c8-870ec330cd19" (UID: "43619c62-7adb-49b0-94c8-870ec330cd19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:32.554706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.553431    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-istio-token" pod "43619c62-7adb-49b0-94c8-870ec330cd19" (UID: "43619c62-7adb-49b0-94c8-870ec330cd19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:32.554844 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.554821    7082 generic.go:356] "Generic (PLEG): container finished" podID="43619c62-7adb-49b0-94c8-870ec330cd19" containerID="00d6c71b54c18baf5f3c04df568e9ac647fd849c24c10f71a5e19b2d308cac46" exitCode=137
Oct 29 22:08:32.554975 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.554956    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2" event={"ID":"43619c62-7adb-49b0-94c8-870ec330cd19","Type":"ContainerDied","Data":"00d6c71b54c18baf5f3c04df568e9ac647fd849c24c10f71a5e19b2d308cac46"}
Oct 29 22:08:32.555091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.555015    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2" event={"ID":"43619c62-7adb-49b0-94c8-870ec330cd19","Type":"ContainerDied","Data":"2d2a4a762461a75f26ccc59be5ed738353dc8132c6757f184fc1153683343506"}
Oct 29 22:08:32.555091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.555047    7082 scope.go:122] "RemoveContainer" containerID="00d6c71b54c18baf5f3c04df568e9ac647fd849c24c10f71a5e19b2d308cac46"
Oct 29 22:08:32.555673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.555218    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"
Oct 29 22:08:32.593924 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.593894    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"]
Oct 29 22:08:32.613338 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.613292    7082 scope.go:122] "RemoveContainer" containerID="00d6c71b54c18baf5f3c04df568e9ac647fd849c24c10f71a5e19b2d308cac46"
Oct 29 22:08:32.613819 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:08:32.613767    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d6c71b54c18baf5f3c04df568e9ac647fd849c24c10f71a5e19b2d308cac46\": container with ID starting with 00d6c71b54c18baf5f3c04df568e9ac647fd849c24c10f71a5e19b2d308cac46 not found: ID does not exist" containerID="00d6c71b54c18baf5f3c04df568e9ac647fd849c24c10f71a5e19b2d308cac46"
Oct 29 22:08:32.614400 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.613813    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d6c71b54c18baf5f3c04df568e9ac647fd849c24c10f71a5e19b2d308cac46"} err="failed to get container status \"00d6c71b54c18baf5f3c04df568e9ac647fd849c24c10f71a5e19b2d308cac46\": rpc error: code = NotFound desc = could not find container \"00d6c71b54c18baf5f3c04df568e9ac647fd849c24c10f71a5e19b2d308cac46\": container with ID starting with 00d6c71b54c18baf5f3c04df568e9ac647fd849c24c10f71a5e19b2d308cac46 not found: ID does not exist"
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.652582    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa3614e-71e6-4d35-9046-8876a080c622-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.652740    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa3614e-71e6-4d35-9046-8876a080c622-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.652784    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.652826    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.652889    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.653053    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.653268    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.653308    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77wbw\" (UniqueName: \"kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-kube-api-access-77wbw\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.653372    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.653546    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.653581    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.653594    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43619c62-7adb-49b0-94c8-870ec330cd19-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.653608    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43619c62-7adb-49b0-94c8-870ec330cd19-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.653636    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtkg5\" (UniqueName: \"kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-kube-api-access-qtkg5\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.653649    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.653661    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.653673    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43619c62-7adb-49b0-94c8-870ec330cd19-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.653686    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43619c62-7adb-49b0-94c8-870ec330cd19-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:32.655120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.655012    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.657352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.655294    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.657352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.655421    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa3614e-71e6-4d35-9046-8876a080c622-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.657352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.656372    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.657352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.656385    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.658749 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.658724    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.658903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.658883    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa3614e-71e6-4d35-9046-8876a080c622-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.718610 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.718542    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.719444 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.719415    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77wbw\" (UniqueName: \"kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-kube-api-access-77wbw\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:32.725341 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.725314    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"]
Oct 29 22:08:32.746270 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.746236    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f6171d-openshift-default-64847464f9-s44m2"]
Oct 29 22:08:32.828207 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:32.828174    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:33.371052 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:33.371024    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"]
Oct 29 22:08:33.387012 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:08:33.386978    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa3614e_71e6_4d35_9046_8876a080c622.slice/crio-3a70b1e96dc346e47b0a2c4d0bb124e385a302f455818e3899e69ce8fbbaf941 WatchSource:0}: Error finding container 3a70b1e96dc346e47b0a2c4d0bb124e385a302f455818e3899e69ce8fbbaf941: Status 404 returned error can't find the container with id 3a70b1e96dc346e47b0a2c4d0bb124e385a302f455818e3899e69ce8fbbaf941
Oct 29 22:08:33.390430 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:33.390395    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:08:33.390522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:33.390466    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:08:33.390522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:33.390498    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:08:33.572895 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:33.572859    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv" event={"ID":"1fa3614e-71e6-4d35-9046-8876a080c622","Type":"ContainerStarted","Data":"4579e74b67898eb26085361037e2e59454d627c36403139e47a7811b3f8f1c18"}
Oct 29 22:08:33.573103 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:33.572925    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv" event={"ID":"1fa3614e-71e6-4d35-9046-8876a080c622","Type":"ContainerStarted","Data":"3a70b1e96dc346e47b0a2c4d0bb124e385a302f455818e3899e69ce8fbbaf941"}
Oct 29 22:08:33.631193 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:33.631080    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv" podStartSLOduration=1.631055199 podStartE2EDuration="1.631055199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:33.622786676 +0000 UTC m=+1286.122828975" watchObservedRunningTime="2026-10-29 22:08:33.631055199 +0000 UTC m=+1286.131097500"
Oct 29 22:08:33.669865 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:33.669823    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43619c62-7adb-49b0-94c8-870ec330cd19" path="/var/lib/kubelet/pods/43619c62-7adb-49b0-94c8-870ec330cd19/volumes"
Oct 29 22:08:33.828389 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:33.828342    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:34.585106 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:34.585070    7082 generic.go:356] "Generic (PLEG): container finished" podID="423fc4e9-6575-4690-9263-83ffff40dfc3" containerID="dc7bd3c00431b323bdc0ba5c5d37e12df664a7bccaed4048dfededf10ab964dd" exitCode=0
Oct 29 22:08:34.585358 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:34.585172    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8" event={"ID":"423fc4e9-6575-4690-9263-83ffff40dfc3","Type":"ContainerDied","Data":"dc7bd3c00431b323bdc0ba5c5d37e12df664a7bccaed4048dfededf10ab964dd"}
Oct 29 22:08:34.585835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:34.585808    7082 scope.go:122] "RemoveContainer" containerID="dc7bd3c00431b323bdc0ba5c5d37e12df664a7bccaed4048dfededf10ab964dd"
Oct 29 22:08:34.587999 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:34.587967    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:08:34.588123 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:34.588042    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:08:34.588123 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:34.588086    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:08:34.830496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:34.830458    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:08:34.831042 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:34.830518    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv" podUID="1fa3614e-71e6-4d35-9046-8876a080c622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 29 22:08:34.980453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:34.980377    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:08:35.051377 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:35.051341    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"]
Oct 29 22:08:35.057793 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:35.057763    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37a3d713-zdmws"]
Oct 29 22:08:35.653790 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:35.653414    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1649f1-1109-40e1-bd32-cc2d6492d4e9" path="/var/lib/kubelet/pods/3f1649f1-1109-40e1-bd32-cc2d6492d4e9/volumes"
Oct 29 22:08:35.682659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:35.677541    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8" event={"ID":"423fc4e9-6575-4690-9263-83ffff40dfc3","Type":"ContainerStarted","Data":"0cd0fda47c37bd337f9da211613b13a3e1350a4e0aa178c0c9db90a6db351dc1"}
Oct 29 22:08:35.843736 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:35.843708    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:36.568410 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:36.568383    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:08:36.573692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:36.573669    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:08:36.684674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:36.684636    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:08:36.684878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:36.684694    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:36.685512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:36.685484    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:08:36.685801 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:36.685782    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:08:39.569300 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:39.569212    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"]
Oct 29 22:08:39.571071 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:39.569511    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy" containerID="cri-o://49e6a240c06c3e7a6accf704fccfaac7ff3690fc731a997b60d5b5a3eee8792e" gracePeriod=30
Oct 29 22:08:39.583756 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:39.583724    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"]
Oct 29 22:08:39.584027 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:39.583998    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy" containerID="cri-o://0cbb201369202d83e2353e9943ce27aba7470bf0bed95613c1e700486fa0d22b" gracePeriod=30
Oct 29 22:08:39.589981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:39.589956    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"]
Oct 29 22:08:39.590186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:39.590159    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy" containerID="cri-o://0d7c44b569eed8da940180306301764490f92e94f88d4556487bb14744964f45" gracePeriod=30
Oct 29 22:08:40.235519 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.235484    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"]
Oct 29 22:08:40.236391 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.236375    7082 cpu_manager.go:414] "containerMap: removing container" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy"
Oct 29 22:08:40.236446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.236392    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1d3eec-cb23-48d7-bc11-bbe55580eee2" containerName="istio-proxy"
Oct 29 22:08:40.239503 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.239487    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"
Oct 29 22:08:40.255496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.255004    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:08:40.255496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.255046    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:08:40.260956 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.260934    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"]
Oct 29 22:08:40.296094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.296071    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:08:40.296241 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.296117    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:08:40.306768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.306744    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkdvq\" (UniqueName: \"kubernetes.io/projected/517d67a8-859e-473c-a71f-b2e83ca9077a-kube-api-access-lkdvq\") pod \"maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9\" (UID: \"517d67a8-859e-473c-a71f-b2e83ca9077a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"
Oct 29 22:08:40.306884 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.306870    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9\" (UID: \"517d67a8-859e-473c-a71f-b2e83ca9077a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"
Oct 29 22:08:40.307037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.307024    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9\" (UID: \"517d67a8-859e-473c-a71f-b2e83ca9077a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"
Oct 29 22:08:40.408886 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.408858    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9\" (UID: \"517d67a8-859e-473c-a71f-b2e83ca9077a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"
Oct 29 22:08:40.411318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.409007    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkdvq\" (UniqueName: \"kubernetes.io/projected/517d67a8-859e-473c-a71f-b2e83ca9077a-kube-api-access-lkdvq\") pod \"maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9\" (UID: \"517d67a8-859e-473c-a71f-b2e83ca9077a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"
Oct 29 22:08:40.411318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.409090    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9\" (UID: \"517d67a8-859e-473c-a71f-b2e83ca9077a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"
Oct 29 22:08:40.411318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.409691    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9\" (UID: \"517d67a8-859e-473c-a71f-b2e83ca9077a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"
Oct 29 22:08:40.411445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.411394    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9\" (UID: \"517d67a8-859e-473c-a71f-b2e83ca9077a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"
Oct 29 22:08:40.435364 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.435343    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkdvq\" (UniqueName: \"kubernetes.io/projected/517d67a8-859e-473c-a71f-b2e83ca9077a-kube-api-access-lkdvq\") pod \"maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9\" (UID: \"517d67a8-859e-473c-a71f-b2e83ca9077a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"
Oct 29 22:08:40.551416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.546644    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b575997cc-zthg8"
Oct 29 22:08:40.552739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.552681    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"
Oct 29 22:08:40.936149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:40.936115    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"]
Oct 29 22:08:41.541094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:41.541060    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-t48hs"
Oct 29 22:08:41.926707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:41.926667    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9" event={"ID":"517d67a8-859e-473c-a71f-b2e83ca9077a","Type":"ContainerStarted","Data":"c73c3f04f18696b2d83d849c15d5a55b63f4ec3b16837e52f577f521ec85cf8e"}
Oct 29 22:08:41.926859 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:41.926737    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9" event={"ID":"517d67a8-859e-473c-a71f-b2e83ca9077a","Type":"ContainerStarted","Data":"5604fbc3355c767a565a1a576e10978353c3c46ab3308f12063253269d58cefa"}
Oct 29 22:08:42.938368 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:42.938333    7082 generic.go:356] "Generic (PLEG): container finished" podID="517d67a8-859e-473c-a71f-b2e83ca9077a" containerID="c73c3f04f18696b2d83d849c15d5a55b63f4ec3b16837e52f577f521ec85cf8e" exitCode=0
Oct 29 22:08:42.938805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:42.938421    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9" event={"ID":"517d67a8-859e-473c-a71f-b2e83ca9077a","Type":"ContainerDied","Data":"c73c3f04f18696b2d83d849c15d5a55b63f4ec3b16837e52f577f521ec85cf8e"}
Oct 29 22:08:43.258316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:43.258289    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"
Oct 29 22:08:43.284163 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:43.284135    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-openshift-ca\") pod \"517d67a8-859e-473c-a71f-b2e83ca9077a\" (UID: \"517d67a8-859e-473c-a71f-b2e83ca9077a\") "
Oct 29 22:08:43.284326 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:43.284231    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-xks-ca\") pod \"517d67a8-859e-473c-a71f-b2e83ca9077a\" (UID: \"517d67a8-859e-473c-a71f-b2e83ca9077a\") "
Oct 29 22:08:43.284408 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:43.284391    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517d67a8-859e-473c-a71f-b2e83ca9077a-kube-api-access-lkdvq\" (UniqueName: \"kubernetes.io/projected/517d67a8-859e-473c-a71f-b2e83ca9077a-kube-api-access-lkdvq\") pod \"517d67a8-859e-473c-a71f-b2e83ca9077a\" (UID: \"517d67a8-859e-473c-a71f-b2e83ca9077a\") "
Oct 29 22:08:43.285279 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:43.284504    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-openshift-ca" pod "517d67a8-859e-473c-a71f-b2e83ca9077a" (UID: "517d67a8-859e-473c-a71f-b2e83ca9077a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:43.285342 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:43.285276    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:43.286804 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:43.286781    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-xks-ca" pod "517d67a8-859e-473c-a71f-b2e83ca9077a" (UID: "517d67a8-859e-473c-a71f-b2e83ca9077a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:08:43.286895 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:43.286822    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517d67a8-859e-473c-a71f-b2e83ca9077a-kube-api-access-lkdvq" pod "517d67a8-859e-473c-a71f-b2e83ca9077a" (UID: "517d67a8-859e-473c-a71f-b2e83ca9077a"). InnerVolumeSpecName "kube-api-access-lkdvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:43.387115 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:43.387077    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkdvq\" (UniqueName: \"kubernetes.io/projected/517d67a8-859e-473c-a71f-b2e83ca9077a-kube-api-access-lkdvq\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:43.387115 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:43.387111    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/517d67a8-859e-473c-a71f-b2e83ca9077a-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:43.949219 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:43.949145    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"
Oct 29 22:08:43.949219 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:43.949177    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9" event={"ID":"517d67a8-859e-473c-a71f-b2e83ca9077a","Type":"ContainerDied","Data":"5604fbc3355c767a565a1a576e10978353c3c46ab3308f12063253269d58cefa"}
Oct 29 22:08:43.949651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:43.949223    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5604fbc3355c767a565a1a576e10978353c3c46ab3308f12063253269d58cefa"
Oct 29 22:08:44.944315 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:44.944287    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:08:44.971043 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:44.971014    7082 generic.go:356] "Generic (PLEG): container finished" podID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerID="0d7c44b569eed8da940180306301764490f92e94f88d4556487bb14744964f45" exitCode=0
Oct 29 22:08:44.971490 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:44.971079    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"
Oct 29 22:08:44.971490 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:44.971179    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" event={"ID":"db6e6557-f68f-423d-bc71-67dcb9f25587","Type":"ContainerDied","Data":"0d7c44b569eed8da940180306301764490f92e94f88d4556487bb14744964f45"}
Oct 29 22:08:44.971490 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:44.971230    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4" event={"ID":"db6e6557-f68f-423d-bc71-67dcb9f25587","Type":"ContainerDied","Data":"e7f6bc7e916ec5cd27edd31333b3720a96f0bdfd664222dc05b83dae1c764f81"}
Oct 29 22:08:44.971490 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:44.971249    7082 scope.go:122] "RemoveContainer" containerID="0d7c44b569eed8da940180306301764490f92e94f88d4556487bb14744964f45"
Oct 29 22:08:44.971733 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:44.971523    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:08:44.974696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:44.974066    7082 generic.go:356] "Generic (PLEG): container finished" podID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerID="49e6a240c06c3e7a6accf704fccfaac7ff3690fc731a997b60d5b5a3eee8792e" exitCode=0
Oct 29 22:08:44.974696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:44.974301    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" event={"ID":"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019","Type":"ContainerDied","Data":"49e6a240c06c3e7a6accf704fccfaac7ff3690fc731a997b60d5b5a3eee8792e"}
Oct 29 22:08:44.977092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:44.977065    7082 generic.go:356] "Generic (PLEG): container finished" podID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerID="0cbb201369202d83e2353e9943ce27aba7470bf0bed95613c1e700486fa0d22b" exitCode=0
Oct 29 22:08:44.977249 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:44.977157    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"
Oct 29 22:08:44.977249 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:44.977230    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq" event={"ID":"93b8ea59-e14b-4658-abea-a9234b68fbb9","Type":"ContainerDied","Data":"0cbb201369202d83e2353e9943ce27aba7470bf0bed95613c1e700486fa0d22b"}
Oct 29 22:08:45.009253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.008440    7082 scope.go:122] "RemoveContainer" containerID="0d7c44b569eed8da940180306301764490f92e94f88d4556487bb14744964f45"
Oct 29 22:08:45.009253 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:08:45.008848    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7c44b569eed8da940180306301764490f92e94f88d4556487bb14744964f45\": container with ID starting with 0d7c44b569eed8da940180306301764490f92e94f88d4556487bb14744964f45 not found: ID does not exist" containerID="0d7c44b569eed8da940180306301764490f92e94f88d4556487bb14744964f45"
Oct 29 22:08:45.009253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.008882    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7c44b569eed8da940180306301764490f92e94f88d4556487bb14744964f45"} err="failed to get container status \"0d7c44b569eed8da940180306301764490f92e94f88d4556487bb14744964f45\": rpc error: code = NotFound desc = could not find container \"0d7c44b569eed8da940180306301764490f92e94f88d4556487bb14744964f45\": container with ID starting with 0d7c44b569eed8da940180306301764490f92e94f88d4556487bb14744964f45 not found: ID does not exist"
Oct 29 22:08:45.009253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.008908    7082 scope.go:122] "RemoveContainer" containerID="0cbb201369202d83e2353e9943ce27aba7470bf0bed95613c1e700486fa0d22b"
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020119    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-token\" (UniqueName: \"kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-token\") pod \"db6e6557-f68f-423d-bc71-67dcb9f25587\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020196    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-socket\") pod \"93b8ea59-e14b-4658-abea-a9234b68fbb9\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020323    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-credential-socket\") pod \"93b8ea59-e14b-4658-abea-a9234b68fbb9\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020416    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-certs\") pod \"db6e6557-f68f-423d-bc71-67dcb9f25587\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020458    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-socket\") pod \"db6e6557-f68f-423d-bc71-67dcb9f25587\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020503    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-data\") pod \"93b8ea59-e14b-4658-abea-a9234b68fbb9\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020536    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-data\") pod \"db6e6557-f68f-423d-bc71-67dcb9f25587\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020573    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b8ea59-e14b-4658-abea-a9234b68fbb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b8ea59-e14b-4658-abea-a9234b68fbb9-istiod-ca-cert\") pod \"93b8ea59-e14b-4658-abea-a9234b68fbb9\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020620    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-envoy\") pod \"93b8ea59-e14b-4658-abea-a9234b68fbb9\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020678    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-credential-socket\") pod \"db6e6557-f68f-423d-bc71-67dcb9f25587\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020728    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db6e6557-f68f-423d-bc71-67dcb9f25587-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6e6557-f68f-423d-bc71-67dcb9f25587-istiod-ca-cert\") pod \"db6e6557-f68f-423d-bc71-67dcb9f25587\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020768    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-certs\") pod \"93b8ea59-e14b-4658-abea-a9234b68fbb9\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020798    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-kube-api-access-626zg\" (UniqueName: \"kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-kube-api-access-626zg\") pod \"db6e6557-f68f-423d-bc71-67dcb9f25587\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020874    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-podinfo\") pod \"db6e6557-f68f-423d-bc71-67dcb9f25587\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.020911    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-envoy\") pod \"db6e6557-f68f-423d-bc71-67dcb9f25587\" (UID: \"db6e6557-f68f-423d-bc71-67dcb9f25587\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.021092    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-kube-api-access-6cgzx\" (UniqueName: \"kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-kube-api-access-6cgzx\") pod \"93b8ea59-e14b-4658-abea-a9234b68fbb9\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.021130    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-token\" (UniqueName: \"kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-token\") pod \"93b8ea59-e14b-4658-abea-a9234b68fbb9\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.021223    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-podinfo\") pod \"93b8ea59-e14b-4658-abea-a9234b68fbb9\" (UID: \"93b8ea59-e14b-4658-abea-a9234b68fbb9\") "
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.023041    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-certs" pod "93b8ea59-e14b-4658-abea-a9234b68fbb9" (UID: "93b8ea59-e14b-4658-abea-a9234b68fbb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.023414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.023243    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-credential-socket" pod "db6e6557-f68f-423d-bc71-67dcb9f25587" (UID: "db6e6557-f68f-423d-bc71-67dcb9f25587"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.026273 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.023437    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-data" pod "93b8ea59-e14b-4658-abea-a9234b68fbb9" (UID: "93b8ea59-e14b-4658-abea-a9234b68fbb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.026273 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.023805    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-certs" pod "db6e6557-f68f-423d-bc71-67dcb9f25587" (UID: "db6e6557-f68f-423d-bc71-67dcb9f25587"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.026273 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.024105    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db6e6557-f68f-423d-bc71-67dcb9f25587-istiod-ca-cert" pod "db6e6557-f68f-423d-bc71-67dcb9f25587" (UID: "db6e6557-f68f-423d-bc71-67dcb9f25587"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:45.026880 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.026762    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-token" pod "db6e6557-f68f-423d-bc71-67dcb9f25587" (UID: "db6e6557-f68f-423d-bc71-67dcb9f25587"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.026975 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.026880    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-socket" pod "db6e6557-f68f-423d-bc71-67dcb9f25587" (UID: "db6e6557-f68f-423d-bc71-67dcb9f25587"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.027415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.027049    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-data" pod "db6e6557-f68f-423d-bc71-67dcb9f25587" (UID: "db6e6557-f68f-423d-bc71-67dcb9f25587"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.027415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.027213    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-socket" pod "93b8ea59-e14b-4658-abea-a9234b68fbb9" (UID: "93b8ea59-e14b-4658-abea-a9234b68fbb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.028294 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.027701    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-credential-socket" pod "93b8ea59-e14b-4658-abea-a9234b68fbb9" (UID: "93b8ea59-e14b-4658-abea-a9234b68fbb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.028294 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.027805    7082 scope.go:122] "RemoveContainer" containerID="867962ef794ba40d2870c2150b83fdac0911ec4e1caae3f4fac181e863f56027"
Oct 29 22:08:45.029510 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.028613    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b8ea59-e14b-4658-abea-a9234b68fbb9-istiod-ca-cert" pod "93b8ea59-e14b-4658-abea-a9234b68fbb9" (UID: "93b8ea59-e14b-4658-abea-a9234b68fbb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:45.034154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.034134    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-kube-api-access-6cgzx" pod "93b8ea59-e14b-4658-abea-a9234b68fbb9" (UID: "93b8ea59-e14b-4658-abea-a9234b68fbb9"). InnerVolumeSpecName "kube-api-access-6cgzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.036070 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.036021    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-token" pod "93b8ea59-e14b-4658-abea-a9234b68fbb9" (UID: "93b8ea59-e14b-4658-abea-a9234b68fbb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.036308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.036267    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-envoy" pod "db6e6557-f68f-423d-bc71-67dcb9f25587" (UID: "db6e6557-f68f-423d-bc71-67dcb9f25587"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.036308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.036294    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-podinfo" pod "db6e6557-f68f-423d-bc71-67dcb9f25587" (UID: "db6e6557-f68f-423d-bc71-67dcb9f25587"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:45.036619 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.036581    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-podinfo" pod "93b8ea59-e14b-4658-abea-a9234b68fbb9" (UID: "93b8ea59-e14b-4658-abea-a9234b68fbb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:45.038960 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.038610    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-kube-api-access-626zg" pod "db6e6557-f68f-423d-bc71-67dcb9f25587" (UID: "db6e6557-f68f-423d-bc71-67dcb9f25587"). InnerVolumeSpecName "kube-api-access-626zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.042417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.041664    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-envoy" pod "93b8ea59-e14b-4658-abea-a9234b68fbb9" (UID: "93b8ea59-e14b-4658-abea-a9234b68fbb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.051222 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.051200    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:08:45.123968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.123606    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-kube-api-access-qllq4\" (UniqueName: \"kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-kube-api-access-qllq4\") pod \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") "
Oct 29 22:08:45.123968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.123712    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-envoy\") pod \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") "
Oct 29 22:08:45.123968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.123780    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-podinfo\") pod \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") "
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.124336    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-data\") pod \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") "
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.124406    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-certs\") pod \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") "
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.124506    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-credential-socket\") pod \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") "
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.124702    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istiod-ca-cert\") pod \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") "
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.124753    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-socket\") pod \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") "
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.124786    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-token\" (UniqueName: \"kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-token\") pod \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\" (UID: \"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019\") "
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125667    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125692    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125707    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125720    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125734    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125748    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125761    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125774    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125786    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b8ea59-e14b-4658-abea-a9234b68fbb9-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125800    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125813    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125826    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db6e6557-f68f-423d-bc71-67dcb9f25587-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125839    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b8ea59-e14b-4658-abea-a9234b68fbb9-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125853    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-626zg\" (UniqueName: \"kubernetes.io/projected/db6e6557-f68f-423d-bc71-67dcb9f25587-kube-api-access-626zg\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125866    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125878    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db6e6557-f68f-423d-bc71-67dcb9f25587-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125892    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cgzx\" (UniqueName: \"kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-kube-api-access-6cgzx\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125906    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b8ea59-e14b-4658-abea-a9234b68fbb9-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.124783    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-certs" pod "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" (UID: "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125013    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-data" pod "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" (UID: "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125405    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-socket" pod "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" (UID: "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.125989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.125682    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istiod-ca-cert" pod "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" (UID: "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:45.127934 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.126982    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-credential-socket" pod "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" (UID: "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.127934 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.127222    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-kube-api-access-qllq4" pod "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" (UID: "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019"). InnerVolumeSpecName "kube-api-access-qllq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.128509 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.128465    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"]
Oct 29 22:08:45.129037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.129012    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-envoy" pod "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" (UID: "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.129806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.129781    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-podinfo" pod "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" (UID: "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:45.130305 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.130276    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-token" pod "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" (UID: "a1d7cee5-7a1c-4bec-a48a-4ddf35e08019"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.144547 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.143011    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c992d3a-tqpnc"]
Oct 29 22:08:45.227984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.227893    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.227984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.227921    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.227984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.227935    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.227984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.227945    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.227984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.227954    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.227984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.227962    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.227984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.227970    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.227984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.227978    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.227984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.227986    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qllq4\" (UniqueName: \"kubernetes.io/projected/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019-kube-api-access-qllq4\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.319334 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.319289    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"]
Oct 29 22:08:45.326003 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.325977    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-7sqb4"]
Oct 29 22:08:45.359151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.359128    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"]
Oct 29 22:08:45.363539 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.363517    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-rzqzq"]
Oct 29 22:08:45.739812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.739704    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2f1615-58ef-4aed-9cee-55ee002b4971" path="/var/lib/kubelet/pods/4f2f1615-58ef-4aed-9cee-55ee002b4971/volumes"
Oct 29 22:08:45.740846 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.740784    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" path="/var/lib/kubelet/pods/93b8ea59-e14b-4658-abea-a9234b68fbb9/volumes"
Oct 29 22:08:45.742412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.742365    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" path="/var/lib/kubelet/pods/db6e6557-f68f-423d-bc71-67dcb9f25587/volumes"
Oct 29 22:08:45.990909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.990824    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"
Oct 29 22:08:45.990909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.990864    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k" event={"ID":"a1d7cee5-7a1c-4bec-a48a-4ddf35e08019","Type":"ContainerDied","Data":"0d8dacb6696d6abbfbe9100db97609a0d09131922c3ba13c0073c49596c4a783"}
Oct 29 22:08:45.992765 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:45.990941    7082 scope.go:122] "RemoveContainer" containerID="49e6a240c06c3e7a6accf704fccfaac7ff3690fc731a997b60d5b5a3eee8792e"
Oct 29 22:08:46.044329 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:46.044304    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"]
Oct 29 22:08:46.056228 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:46.056203    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-vn95k"]
Oct 29 22:08:47.169584 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:47.169554    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"]
Oct 29 22:08:47.170046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:47.169789    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4" podUID="9bc1d583-b4dc-4d4b-99c4-c08a945cdad9" containerName="payload-processing" containerID="cri-o://50729da47df72247a6059b135606b119541341d6b11adfee83496c21f732b5ec" gracePeriod=30
Oct 29 22:08:47.217885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:47.217850    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"]
Oct 29 22:08:47.218131 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:47.218104    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4" podUID="17f87fff-d6eb-4aa9-b6b5-cac68c2fed28" containerName="payload-pre-processing" containerID="cri-o://c41051adac08c588f4cecb0bc345afa178be152f9e6fd9a812e7aa3f48d771f0" gracePeriod=30
Oct 29 22:08:47.814714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:47.804410    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" path="/var/lib/kubelet/pods/a1d7cee5-7a1c-4bec-a48a-4ddf35e08019/volumes"
Oct 29 22:08:48.067529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.046233    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"
Oct 29 22:08:48.106141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.106106    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"]
Oct 29 22:08:48.125344 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.125306    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"
Oct 29 22:08:48.127942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.127918    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-734480-9ht4n"]
Oct 29 22:08:48.129004 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.128979    7082 generic.go:356] "Generic (PLEG): container finished" podID="9bc1d583-b4dc-4d4b-99c4-c08a945cdad9" containerID="50729da47df72247a6059b135606b119541341d6b11adfee83496c21f732b5ec" exitCode=0
Oct 29 22:08:48.129195 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.129175    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4" event={"ID":"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9","Type":"ContainerDied","Data":"50729da47df72247a6059b135606b119541341d6b11adfee83496c21f732b5ec"}
Oct 29 22:08:48.129271 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.129257    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4" event={"ID":"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9","Type":"ContainerDied","Data":"2f58e75e32e0be0be5491e427951be7691bfd302a23aa82ab854dab57827dc7d"}
Oct 29 22:08:48.129326 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.129287    7082 scope.go:122] "RemoveContainer" containerID="50729da47df72247a6059b135606b119541341d6b11adfee83496c21f732b5ec"
Oct 29 22:08:48.129429 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.129416    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"
Oct 29 22:08:48.132434 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.132402    7082 generic.go:356] "Generic (PLEG): container finished" podID="17f87fff-d6eb-4aa9-b6b5-cac68c2fed28" containerID="c41051adac08c588f4cecb0bc345afa178be152f9e6fd9a812e7aa3f48d771f0" exitCode=0
Oct 29 22:08:48.132689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.132564    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4" event={"ID":"17f87fff-d6eb-4aa9-b6b5-cac68c2fed28","Type":"ContainerDied","Data":"c41051adac08c588f4cecb0bc345afa178be152f9e6fd9a812e7aa3f48d771f0"}
Oct 29 22:08:48.132927 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.132912    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"
Oct 29 22:08:48.148384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.148360    7082 scope.go:122] "RemoveContainer" containerID="50729da47df72247a6059b135606b119541341d6b11adfee83496c21f732b5ec"
Oct 29 22:08:48.148776 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:08:48.148755    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50729da47df72247a6059b135606b119541341d6b11adfee83496c21f732b5ec\": container with ID starting with 50729da47df72247a6059b135606b119541341d6b11adfee83496c21f732b5ec not found: ID does not exist" containerID="50729da47df72247a6059b135606b119541341d6b11adfee83496c21f732b5ec"
Oct 29 22:08:48.148864 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.148788    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50729da47df72247a6059b135606b119541341d6b11adfee83496c21f732b5ec"} err="failed to get container status \"50729da47df72247a6059b135606b119541341d6b11adfee83496c21f732b5ec\": rpc error: code = NotFound desc = could not find container \"50729da47df72247a6059b135606b119541341d6b11adfee83496c21f732b5ec\": container with ID starting with 50729da47df72247a6059b135606b119541341d6b11adfee83496c21f732b5ec not found: ID does not exist"
Oct 29 22:08:48.148864 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.148812    7082 scope.go:122] "RemoveContainer" containerID="c41051adac08c588f4cecb0bc345afa178be152f9e6fd9a812e7aa3f48d771f0"
Oct 29 22:08:48.172053 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.172020    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-kube-api-access-ng7cx\" (UniqueName: \"kubernetes.io/projected/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-kube-api-access-ng7cx\") pod \"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9\" (UID: \"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9\") "
Oct 29 22:08:48.172508 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.172487    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-plugins-config-volume\") pod \"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9\" (UID: \"9bc1d583-b4dc-4d4b-99c4-c08a945cdad9\") "
Oct 29 22:08:48.173179 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.173147    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-plugins-config-volume" pod "9bc1d583-b4dc-4d4b-99c4-c08a945cdad9" (UID: "9bc1d583-b4dc-4d4b-99c4-c08a945cdad9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:48.173373 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.173354    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:48.174329 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.174309    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-kube-api-access-ng7cx" pod "9bc1d583-b4dc-4d4b-99c4-c08a945cdad9" (UID: "9bc1d583-b4dc-4d4b-99c4-c08a945cdad9"). InnerVolumeSpecName "kube-api-access-ng7cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:48.343704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.341131    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-plugins-config-volume\") pod \"17f87fff-d6eb-4aa9-b6b5-cac68c2fed28\" (UID: \"17f87fff-d6eb-4aa9-b6b5-cac68c2fed28\") "
Oct 29 22:08:48.343704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.341261    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-kube-api-access-qrsj2\" (UniqueName: \"kubernetes.io/projected/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-kube-api-access-qrsj2\") pod \"17f87fff-d6eb-4aa9-b6b5-cac68c2fed28\" (UID: \"17f87fff-d6eb-4aa9-b6b5-cac68c2fed28\") "
Oct 29 22:08:48.343704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.342177    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-plugins-config-volume" pod "17f87fff-d6eb-4aa9-b6b5-cac68c2fed28" (UID: "17f87fff-d6eb-4aa9-b6b5-cac68c2fed28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:48.343704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.342786    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:48.343704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.342815    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng7cx\" (UniqueName: \"kubernetes.io/projected/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9-kube-api-access-ng7cx\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:48.349147 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.346180    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-kube-api-access-qrsj2" pod "17f87fff-d6eb-4aa9-b6b5-cac68c2fed28" (UID: "17f87fff-d6eb-4aa9-b6b5-cac68c2fed28"). InnerVolumeSpecName "kube-api-access-qrsj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:48.471971 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.471185    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrsj2\" (UniqueName: \"kubernetes.io/projected/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28-kube-api-access-qrsj2\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:48.524575 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.524535    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"]
Oct 29 22:08:48.533257 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.533226    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-734480-66479fd5db-ct8c4"]
Oct 29 22:08:48.563178 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.563149    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"]
Oct 29 22:08:48.569042 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:48.569017    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-734480-8477f98fc9-9wrn4"]
Oct 29 22:08:49.752165 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:49.752130    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f87fff-d6eb-4aa9-b6b5-cac68c2fed28" path="/var/lib/kubelet/pods/17f87fff-d6eb-4aa9-b6b5-cac68c2fed28/volumes"
Oct 29 22:08:49.752864 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:49.752848    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc1d583-b4dc-4d4b-99c4-c08a945cdad9" path="/var/lib/kubelet/pods/9bc1d583-b4dc-4d4b-99c4-c08a945cdad9/volumes"
Oct 29 22:08:49.753413 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:49.753397    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce8bea9-8a70-4e28-9d56-53a0932d1b93" path="/var/lib/kubelet/pods/9ce8bea9-8a70-4e28-9d56-53a0932d1b93/volumes"
Oct 29 22:08:50.683093 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:50.683064    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"]
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.737335    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"]
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.738533    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9bc1d583-b4dc-4d4b-99c4-c08a945cdad9" containerName="payload-processing"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.738554    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc1d583-b4dc-4d4b-99c4-c08a945cdad9" containerName="payload-processing"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.738568    7082 cpu_manager.go:414] "containerMap: removing container" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.738577    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.738599    7082 cpu_manager.go:414] "containerMap: removing container" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.738607    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.738658    7082 cpu_manager.go:414] "containerMap: removing container" podUID="517d67a8-859e-473c-a71f-b2e83ca9077a" containerName="revoke-keys"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.738667    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517d67a8-859e-473c-a71f-b2e83ca9077a" containerName="revoke-keys"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.738706    7082 cpu_manager.go:414] "containerMap: removing container" podUID="17f87fff-d6eb-4aa9-b6b5-cac68c2fed28" containerName="payload-pre-processing"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.738713    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f87fff-d6eb-4aa9-b6b5-cac68c2fed28" containerName="payload-pre-processing"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.738746    7082 cpu_manager.go:414] "containerMap: removing container" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.738755    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.739432    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="db6e6557-f68f-423d-bc71-67dcb9f25587" containerName="istio-proxy"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.739459    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="517d67a8-859e-473c-a71f-b2e83ca9077a" containerName="revoke-keys"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.739474    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.739486    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f87fff-d6eb-4aa9-b6b5-cac68c2fed28" containerName="payload-pre-processing"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.739500    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d7cee5-7a1c-4bec-a48a-4ddf35e08019" containerName="istio-proxy"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.739510    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc1d583-b4dc-4d4b-99c4-c08a945cdad9" containerName="payload-processing"
Oct 29 22:08:53.746685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.739519    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy"
Oct 29 22:08:53.749151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.749126    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"
Oct 29 22:08:53.767856 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.765516    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"]
Oct 29 22:08:53.770286 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.769287    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-edc2d3\""
Oct 29 22:08:53.770286 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.769355    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-edc2d3-dockercfg-v6crb\""
Oct 29 22:08:53.782113 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.782084    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg\" (UID: \"55bc3608-7cb5-4874-9ffa-7e62c0f8188b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"
Oct 29 22:08:53.782342 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.782321    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbvs\" (UniqueName: \"kubernetes.io/projected/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-kube-api-access-5qbvs\") pod \"payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg\" (UID: \"55bc3608-7cb5-4874-9ffa-7e62c0f8188b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"
Oct 29 22:08:53.902840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.902787    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg\" (UID: \"55bc3608-7cb5-4874-9ffa-7e62c0f8188b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"
Oct 29 22:08:53.911814 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.911766    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbvs\" (UniqueName: \"kubernetes.io/projected/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-kube-api-access-5qbvs\") pod \"payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg\" (UID: \"55bc3608-7cb5-4874-9ffa-7e62c0f8188b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"
Oct 29 22:08:53.999706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:53.949789    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg\" (UID: \"55bc3608-7cb5-4874-9ffa-7e62c0f8188b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"
Oct 29 22:08:54.054104 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:54.054056    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbvs\" (UniqueName: \"kubernetes.io/projected/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-kube-api-access-5qbvs\") pod \"payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg\" (UID: \"55bc3608-7cb5-4874-9ffa-7e62c0f8188b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"
Oct 29 22:08:54.110679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:54.091304    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"
Oct 29 22:08:54.462662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:54.459752    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"]
Oct 29 22:08:54.462662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:54.461017    7082 cpu_manager.go:414] "containerMap: removing container" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy"
Oct 29 22:08:54.462662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:54.461038    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b8ea59-e14b-4658-abea-a9234b68fbb9" containerName="istio-proxy"
Oct 29 22:08:54.466278 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:54.466259    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"
Oct 29 22:08:54.474617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:54.474594    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"]
Oct 29 22:08:54.664006 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:54.663946    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlx9z\" (UniqueName: \"kubernetes.io/projected/e98aa080-2663-42a6-b231-3b49fe819e26-kube-api-access-hlx9z\") pod \"payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf\" (UID: \"e98aa080-2663-42a6-b231-3b49fe819e26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"
Oct 29 22:08:54.748285 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:54.748195    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98aa080-2663-42a6-b231-3b49fe819e26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf\" (UID: \"e98aa080-2663-42a6-b231-3b49fe819e26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"
Oct 29 22:08:55.210651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:55.198412    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98aa080-2663-42a6-b231-3b49fe819e26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf\" (UID: \"e98aa080-2663-42a6-b231-3b49fe819e26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"
Oct 29 22:08:55.273146 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:55.249047    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98aa080-2663-42a6-b231-3b49fe819e26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf\" (UID: \"e98aa080-2663-42a6-b231-3b49fe819e26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"
Oct 29 22:08:55.273365 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:55.273329    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlx9z\" (UniqueName: \"kubernetes.io/projected/e98aa080-2663-42a6-b231-3b49fe819e26-kube-api-access-hlx9z\") pod \"payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf\" (UID: \"e98aa080-2663-42a6-b231-3b49fe819e26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"
Oct 29 22:08:55.448651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:55.438710    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlx9z\" (UniqueName: \"kubernetes.io/projected/e98aa080-2663-42a6-b231-3b49fe819e26-kube-api-access-hlx9z\") pod \"payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf\" (UID: \"e98aa080-2663-42a6-b231-3b49fe819e26\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"
Oct 29 22:08:55.473054 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:55.472956    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"
Oct 29 22:08:55.683458 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:55.683313    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk" podUID="e07e2d2d-0845-4243-82ac-6a17427f2290" containerName="payload-processing" containerID="cri-o://443cc8e317eca3d08006e70287bb0474fd0c0b31a4b845849a34d4e8b33d0843" gracePeriod=25
Oct 29 22:08:55.717706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:55.717463    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"]
Oct 29 22:08:56.499280 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:56.462494    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"]
Oct 29 22:08:56.591712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:56.584902    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"]
Oct 29 22:08:56.591712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:56.585142    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8" podUID="423fc4e9-6575-4690-9263-83ffff40dfc3" containerName="istio-proxy" containerID="cri-o://0cd0fda47c37bd337f9da211613b13a3e1350a4e0aa178c0c9db90a6db351dc1" gracePeriod=30
Oct 29 22:08:57.020667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:56.994877    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"
Oct 29 22:08:57.141878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.135874    7082 generic.go:356] "Generic (PLEG): container finished" podID="e07e2d2d-0845-4243-82ac-6a17427f2290" containerID="443cc8e317eca3d08006e70287bb0474fd0c0b31a4b845849a34d4e8b33d0843" exitCode=0
Oct 29 22:08:57.141878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.136025    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk" event={"ID":"e07e2d2d-0845-4243-82ac-6a17427f2290","Type":"ContainerDied","Data":"443cc8e317eca3d08006e70287bb0474fd0c0b31a4b845849a34d4e8b33d0843"}
Oct 29 22:08:57.141878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.136091    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk" event={"ID":"e07e2d2d-0845-4243-82ac-6a17427f2290","Type":"ContainerDied","Data":"8d1872e675db5c2d5bb84995fe4730f66a01fcff34bffdec60c7c7030596d980"}
Oct 29 22:08:57.141878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.136127    7082 scope.go:122] "RemoveContainer" containerID="443cc8e317eca3d08006e70287bb0474fd0c0b31a4b845849a34d4e8b33d0843"
Oct 29 22:08:57.141878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.136322    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"
Oct 29 22:08:57.187868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.179745    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"]
Oct 29 22:08:57.187868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.181049    7082 cpu_manager.go:414] "containerMap: removing container" podUID="e07e2d2d-0845-4243-82ac-6a17427f2290" containerName="payload-processing"
Oct 29 22:08:57.187868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.181069    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07e2d2d-0845-4243-82ac-6a17427f2290" containerName="payload-processing"
Oct 29 22:08:57.187868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.183854    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07e2d2d-0845-4243-82ac-6a17427f2290" containerName="payload-processing"
Oct 29 22:08:57.199812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.198686    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"
Oct 29 22:08:57.245750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.216462    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:08:57.245750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.223231    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:08:57.245750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.226452    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07e2d2d-0845-4243-82ac-6a17427f2290-kube-api-access-9cjkt\" (UniqueName: \"kubernetes.io/projected/e07e2d2d-0845-4243-82ac-6a17427f2290-kube-api-access-9cjkt\") pod \"e07e2d2d-0845-4243-82ac-6a17427f2290\" (UID: \"e07e2d2d-0845-4243-82ac-6a17427f2290\") "
Oct 29 22:08:57.245750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.226519    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e07e2d2d-0845-4243-82ac-6a17427f2290-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07e2d2d-0845-4243-82ac-6a17427f2290-plugins-config-volume\") pod \"e07e2d2d-0845-4243-82ac-6a17427f2290\" (UID: \"e07e2d2d-0845-4243-82ac-6a17427f2290\") "
Oct 29 22:08:57.245750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.228484    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf" event={"ID":"e98aa080-2663-42a6-b231-3b49fe819e26","Type":"ContainerStarted","Data":"60ae96ff627078de43c4641023474b035820d507b69f08219de1b3962267896a"}
Oct 29 22:08:57.245750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.229034    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07e2d2d-0845-4243-82ac-6a17427f2290-plugins-config-volume" pod "e07e2d2d-0845-4243-82ac-6a17427f2290" (UID: "e07e2d2d-0845-4243-82ac-6a17427f2290"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:57.245750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.233392    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg" event={"ID":"55bc3608-7cb5-4874-9ffa-7e62c0f8188b","Type":"ContainerStarted","Data":"10114397007db35351366bf6c9501f8f5c35a4e9fc358a2f9abc38b5e31ea927"}
Oct 29 22:08:57.245750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.233444    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg" event={"ID":"55bc3608-7cb5-4874-9ffa-7e62c0f8188b","Type":"ContainerStarted","Data":"40e4f111e86e20afadf3f5024b6514e3166236fec0ef516ff8be1378e2eee461"}
Oct 29 22:08:57.245750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.234123    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"
Oct 29 22:08:57.261718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.261691    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"]
Oct 29 22:08:57.273585 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.273558    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:08:57.273734 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.273713    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:08:57.274383 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.274353    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07e2d2d-0845-4243-82ac-6a17427f2290-kube-api-access-9cjkt" pod "e07e2d2d-0845-4243-82ac-6a17427f2290" (UID: "e07e2d2d-0845-4243-82ac-6a17427f2290"). InnerVolumeSpecName "kube-api-access-9cjkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:57.332830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.330102    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp\" (UID: \"60fe150f-6b50-4fb2-baec-6582ba520f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"
Oct 29 22:08:57.332830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.330206    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp\" (UID: \"60fe150f-6b50-4fb2-baec-6582ba520f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"
Oct 29 22:08:57.332830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.330404    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g5jk\" (UniqueName: \"kubernetes.io/projected/60fe150f-6b50-4fb2-baec-6582ba520f74-kube-api-access-9g5jk\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp\" (UID: \"60fe150f-6b50-4fb2-baec-6582ba520f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"
Oct 29 22:08:57.332830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.330566    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cjkt\" (UniqueName: \"kubernetes.io/projected/e07e2d2d-0845-4243-82ac-6a17427f2290-kube-api-access-9cjkt\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:57.332830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.330583    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07e2d2d-0845-4243-82ac-6a17427f2290-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:08:57.350755 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.349281    7082 scope.go:122] "RemoveContainer" containerID="443cc8e317eca3d08006e70287bb0474fd0c0b31a4b845849a34d4e8b33d0843"
Oct 29 22:08:57.350755 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:08:57.350195    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443cc8e317eca3d08006e70287bb0474fd0c0b31a4b845849a34d4e8b33d0843\": container with ID starting with 443cc8e317eca3d08006e70287bb0474fd0c0b31a4b845849a34d4e8b33d0843 not found: ID does not exist" containerID="443cc8e317eca3d08006e70287bb0474fd0c0b31a4b845849a34d4e8b33d0843"
Oct 29 22:08:57.350755 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.350233    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443cc8e317eca3d08006e70287bb0474fd0c0b31a4b845849a34d4e8b33d0843"} err="failed to get container status \"443cc8e317eca3d08006e70287bb0474fd0c0b31a4b845849a34d4e8b33d0843\": rpc error: code = NotFound desc = could not find container \"443cc8e317eca3d08006e70287bb0474fd0c0b31a4b845849a34d4e8b33d0843\": container with ID starting with 443cc8e317eca3d08006e70287bb0474fd0c0b31a4b845849a34d4e8b33d0843 not found: ID does not exist"
Oct 29 22:08:57.441706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.439393    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp\" (UID: \"60fe150f-6b50-4fb2-baec-6582ba520f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"
Oct 29 22:08:57.441706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.439525    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp\" (UID: \"60fe150f-6b50-4fb2-baec-6582ba520f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"
Oct 29 22:08:57.449651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.446598    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp\" (UID: \"60fe150f-6b50-4fb2-baec-6582ba520f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"
Oct 29 22:08:57.468651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.467105    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp\" (UID: \"60fe150f-6b50-4fb2-baec-6582ba520f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"
Oct 29 22:08:57.472653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.439720    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g5jk\" (UniqueName: \"kubernetes.io/projected/60fe150f-6b50-4fb2-baec-6582ba520f74-kube-api-access-9g5jk\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp\" (UID: \"60fe150f-6b50-4fb2-baec-6582ba520f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"
Oct 29 22:08:57.751696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.750969    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g5jk\" (UniqueName: \"kubernetes.io/projected/60fe150f-6b50-4fb2-baec-6582ba520f74-kube-api-access-9g5jk\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp\" (UID: \"60fe150f-6b50-4fb2-baec-6582ba520f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"
Oct 29 22:08:57.841647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.837676    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg" podStartSLOduration=4.83765703 podStartE2EDuration="4.83765703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:57.307002856 +0000 UTC m=+1309.807045156" watchObservedRunningTime="2026-10-29 22:08:57.83765703 +0000 UTC m=+1310.337699330"
Oct 29 22:08:57.841647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.839700    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"]
Oct 29 22:08:57.848128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.847526    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-301fcc-67f67c7d44-fbvfk"]
Oct 29 22:08:57.956651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:57.956142    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"
Oct 29 22:08:58.054704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.048972    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"]
Oct 29 22:08:58.114694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.102144    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"
Oct 29 22:08:58.124261 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.124205    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-df42c7\""
Oct 29 22:08:58.124425 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.124356    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-df42c7-dockercfg-xbxtg\""
Oct 29 22:08:58.219189 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.219037    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"]
Oct 29 22:08:58.287712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.249471    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2k75\" (UniqueName: \"kubernetes.io/projected/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-kube-api-access-k2k75\") pod \"payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf\" (UID: \"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"
Oct 29 22:08:58.287712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.249694    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf\" (UID: \"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"
Oct 29 22:08:58.524461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.504877    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"]
Oct 29 22:08:58.560097 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.557699    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"
Oct 29 22:08:58.585965 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.583263    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2k75\" (UniqueName: \"kubernetes.io/projected/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-kube-api-access-k2k75\") pod \"payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf\" (UID: \"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"
Oct 29 22:08:58.585965 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.583435    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf\" (UID: \"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"
Oct 29 22:08:58.590135 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.587405    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"]
Oct 29 22:08:58.631707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.631651    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf" event={"ID":"e98aa080-2663-42a6-b231-3b49fe819e26","Type":"ContainerStarted","Data":"ab87208626bc016e35afea7d6b4e6763c06738177822d323501b0c665caed1ab"}
Oct 29 22:08:58.631876 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.631728    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"
Oct 29 22:08:58.632586 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.632536    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf\" (UID: \"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"
Oct 29 22:08:58.660101 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.649703    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2k75\" (UniqueName: \"kubernetes.io/projected/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-kube-api-access-k2k75\") pod \"payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf\" (UID: \"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"
Oct 29 22:08:58.675512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.675462    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf" podStartSLOduration=5.675445974 podStartE2EDuration="5.675445974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:58.67436888 +0000 UTC m=+1311.174411190" watchObservedRunningTime="2026-10-29 22:08:58.675445974 +0000 UTC m=+1311.175488272"
Oct 29 22:08:58.679281 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:08:58.679241    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:08:58.685467 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.685435    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz5c8\" (UniqueName: \"kubernetes.io/projected/995d93a0-d033-41c4-a7a0-1ca49c98efaf-kube-api-access-nz5c8\") pod \"payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z\" (UID: \"995d93a0-d033-41c4-a7a0-1ca49c98efaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"
Oct 29 22:08:58.685697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.685675    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995d93a0-d033-41c4-a7a0-1ca49c98efaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z\" (UID: \"995d93a0-d033-41c4-a7a0-1ca49c98efaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"
Oct 29 22:08:58.778761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.778682    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"
Oct 29 22:08:58.985674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.981919    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995d93a0-d033-41c4-a7a0-1ca49c98efaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z\" (UID: \"995d93a0-d033-41c4-a7a0-1ca49c98efaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"
Oct 29 22:08:58.985674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.982198    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz5c8\" (UniqueName: \"kubernetes.io/projected/995d93a0-d033-41c4-a7a0-1ca49c98efaf-kube-api-access-nz5c8\") pod \"payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z\" (UID: \"995d93a0-d033-41c4-a7a0-1ca49c98efaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"
Oct 29 22:08:58.985674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:58.983408    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995d93a0-d033-41c4-a7a0-1ca49c98efaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z\" (UID: \"995d93a0-d033-41c4-a7a0-1ca49c98efaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"
Oct 29 22:08:59.147595 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:59.121493    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz5c8\" (UniqueName: \"kubernetes.io/projected/995d93a0-d033-41c4-a7a0-1ca49c98efaf-kube-api-access-nz5c8\") pod \"payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z\" (UID: \"995d93a0-d033-41c4-a7a0-1ca49c98efaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"
Oct 29 22:08:59.147595 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:59.134162    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"]
Oct 29 22:08:59.231923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:59.231067    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"
Oct 29 22:08:59.644734 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:59.644594    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07e2d2d-0845-4243-82ac-6a17427f2290" path="/var/lib/kubelet/pods/e07e2d2d-0845-4243-82ac-6a17427f2290/volumes"
Oct 29 22:08:59.645175 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:59.645113    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"]
Oct 29 22:08:59.647058 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:59.647030    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp" event={"ID":"60fe150f-6b50-4fb2-baec-6582ba520f74","Type":"ContainerStarted","Data":"6aa5064c166e5b662610dadb3d00c24dcad7630d3914e15d00b23fc16bd40fee"}
Oct 29 22:08:59.662678 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:08:59.662650    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"]
Oct 29 22:08:59.672466 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:08:59.672443    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995d93a0_d033_41c4_a7a0_1ca49c98efaf.slice/crio-466cfca416ed8e7084401ad0f67c81f0aa3f4e1733262ca856fa85bcc5b2edcc WatchSource:0}: Error finding container 466cfca416ed8e7084401ad0f67c81f0aa3f4e1733262ca856fa85bcc5b2edcc: Status 404 returned error can't find the container with id 466cfca416ed8e7084401ad0f67c81f0aa3f4e1733262ca856fa85bcc5b2edcc
Oct 29 22:09:00.045736 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:00.045702    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"]
Oct 29 22:09:00.051589 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:00.051557    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a15f6f1-f96b2"]
Oct 29 22:09:00.665078 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:00.664978    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf" event={"ID":"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1","Type":"ContainerStarted","Data":"08866d852f549d871dd2cc2e7570be2723ac6e4751e0cd846165a63ec6be6243"}
Oct 29 22:09:00.665253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:00.665100    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf" event={"ID":"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1","Type":"ContainerStarted","Data":"ae7dc48ac9af29cf9a8c2177bbfc69b2541101371bbec89a4b51a28afbf23759"}
Oct 29 22:09:00.665253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:00.665151    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"
Oct 29 22:09:00.669648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:00.668332    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z" event={"ID":"995d93a0-d033-41c4-a7a0-1ca49c98efaf","Type":"ContainerStarted","Data":"d5c80373c4035af159219c87f339ef11f4025fdfaa1971c506692751629119f1"}
Oct 29 22:09:00.669648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:00.668379    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z" event={"ID":"995d93a0-d033-41c4-a7a0-1ca49c98efaf","Type":"ContainerStarted","Data":"466cfca416ed8e7084401ad0f67c81f0aa3f4e1733262ca856fa85bcc5b2edcc"}
Oct 29 22:09:00.669648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:00.669208    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"
Oct 29 22:09:00.671621 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:00.671598    7082 generic.go:356] "Generic (PLEG): container finished" podID="60fe150f-6b50-4fb2-baec-6582ba520f74" containerID="acc1540c6c27921b850c8a0335cf4c9e16dc7576fe8e6b527cb3c681cff18f35" exitCode=0
Oct 29 22:09:00.671889 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:00.671866    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp" event={"ID":"60fe150f-6b50-4fb2-baec-6582ba520f74","Type":"ContainerDied","Data":"acc1540c6c27921b850c8a0335cf4c9e16dc7576fe8e6b527cb3c681cff18f35"}
Oct 29 22:09:00.710815 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:00.710740    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf" podStartSLOduration=3.7106204 podStartE2EDuration="3.7106204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:00.707545925 +0000 UTC m=+1313.207588226" watchObservedRunningTime="2026-10-29 22:09:00.7106204 +0000 UTC m=+1313.210662701"
Oct 29 22:09:01.641119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:01.641080    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e31289b-2e61-4248-85d4-7578914b2cf7" path="/var/lib/kubelet/pods/7e31289b-2e61-4248-85d4-7578914b2cf7/volumes"
Oct 29 22:09:01.958872 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:01.958838    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"
Oct 29 22:09:01.991148 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:01.991104    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60fe150f-6b50-4fb2-baec-6582ba520f74-kube-api-access-9g5jk\" (UniqueName: \"kubernetes.io/projected/60fe150f-6b50-4fb2-baec-6582ba520f74-kube-api-access-9g5jk\") pod \"60fe150f-6b50-4fb2-baec-6582ba520f74\" (UID: \"60fe150f-6b50-4fb2-baec-6582ba520f74\") "
Oct 29 22:09:01.991761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:01.991383    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-openshift-ca\") pod \"60fe150f-6b50-4fb2-baec-6582ba520f74\" (UID: \"60fe150f-6b50-4fb2-baec-6582ba520f74\") "
Oct 29 22:09:01.991761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:01.991565    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-xks-ca\") pod \"60fe150f-6b50-4fb2-baec-6582ba520f74\" (UID: \"60fe150f-6b50-4fb2-baec-6582ba520f74\") "
Oct 29 22:09:01.992311 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:01.992053    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-openshift-ca" pod "60fe150f-6b50-4fb2-baec-6582ba520f74" (UID: "60fe150f-6b50-4fb2-baec-6582ba520f74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:01.992605 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:01.992573    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:02.026644 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:02.010661    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60fe150f-6b50-4fb2-baec-6582ba520f74-kube-api-access-9g5jk" pod "60fe150f-6b50-4fb2-baec-6582ba520f74" (UID: "60fe150f-6b50-4fb2-baec-6582ba520f74"). InnerVolumeSpecName "kube-api-access-9g5jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:02.026644 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:02.016501    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-xks-ca" pod "60fe150f-6b50-4fb2-baec-6582ba520f74" (UID: "60fe150f-6b50-4fb2-baec-6582ba520f74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:09:02.026644 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:02.022788    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z" podStartSLOduration=4.022770349 podStartE2EDuration="4.022770349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:00.799510291 +0000 UTC m=+1313.299552596" watchObservedRunningTime="2026-10-29 22:09:02.022770349 +0000 UTC m=+1314.522812650"
Oct 29 22:09:02.094576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:02.094537    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60fe150f-6b50-4fb2-baec-6582ba520f74-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:02.094576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:02.094570    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g5jk\" (UniqueName: \"kubernetes.io/projected/60fe150f-6b50-4fb2-baec-6582ba520f74-kube-api-access-9g5jk\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:02.707021 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:02.705177    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"
Oct 29 22:09:02.707021 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:02.705702    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp" event={"ID":"60fe150f-6b50-4fb2-baec-6582ba520f74","Type":"ContainerDied","Data":"6aa5064c166e5b662610dadb3d00c24dcad7630d3914e15d00b23fc16bd40fee"}
Oct 29 22:09:02.707021 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:02.705791    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6aa5064c166e5b662610dadb3d00c24dcad7630d3914e15d00b23fc16bd40fee"
Oct 29 22:09:06.716722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:06.707341    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 29 22:09:06.716722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:06.707427    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8" podUID="423fc4e9-6575-4690-9263-83ffff40dfc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 29 22:09:06.938961 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:06.938918    7082 generic.go:356] "Generic (PLEG): container finished" podID="423fc4e9-6575-4690-9263-83ffff40dfc3" containerID="0cd0fda47c37bd337f9da211613b13a3e1350a4e0aa178c0c9db90a6db351dc1" exitCode=0
Oct 29 22:09:06.939172 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:06.939089    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8" event={"ID":"423fc4e9-6575-4690-9263-83ffff40dfc3","Type":"ContainerDied","Data":"0cd0fda47c37bd337f9da211613b13a3e1350a4e0aa178c0c9db90a6db351dc1"}
Oct 29 22:09:06.939304 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:06.939190    7082 scope.go:122] "RemoveContainer" containerID="dc7bd3c00431b323bdc0ba5c5d37e12df664a7bccaed4048dfededf10ab964dd"
Oct 29 22:09:07.082347 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.081990    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"]
Oct 29 22:09:07.155408 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.154149    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc875add-czt7c"]
Oct 29 22:09:07.484450 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.484423    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:09:07.564076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.562606    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-data\") pod \"423fc4e9-6575-4690-9263-83ffff40dfc3\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") "
Oct 29 22:09:07.564076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.562672    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-credential-socket\") pod \"423fc4e9-6575-4690-9263-83ffff40dfc3\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") "
Oct 29 22:09:07.564076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.562726    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-podinfo\") pod \"423fc4e9-6575-4690-9263-83ffff40dfc3\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") "
Oct 29 22:09:07.564076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.562753    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-envoy\") pod \"423fc4e9-6575-4690-9263-83ffff40dfc3\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") "
Oct 29 22:09:07.564076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.562824    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-socket\") pod \"423fc4e9-6575-4690-9263-83ffff40dfc3\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") "
Oct 29 22:09:07.564076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.562884    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-token\" (UniqueName: \"kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-token\") pod \"423fc4e9-6575-4690-9263-83ffff40dfc3\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") "
Oct 29 22:09:07.564076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.562961    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-kube-api-access-4656f\" (UniqueName: \"kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-kube-api-access-4656f\") pod \"423fc4e9-6575-4690-9263-83ffff40dfc3\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") "
Oct 29 22:09:07.564076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.563056    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-certs\") pod \"423fc4e9-6575-4690-9263-83ffff40dfc3\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") "
Oct 29 22:09:07.564076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.563159    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423fc4e9-6575-4690-9263-83ffff40dfc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423fc4e9-6575-4690-9263-83ffff40dfc3-istiod-ca-cert\") pod \"423fc4e9-6575-4690-9263-83ffff40dfc3\" (UID: \"423fc4e9-6575-4690-9263-83ffff40dfc3\") "
Oct 29 22:09:07.564737 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.564345    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-socket" pod "423fc4e9-6575-4690-9263-83ffff40dfc3" (UID: "423fc4e9-6575-4690-9263-83ffff40dfc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:07.565212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.565045    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-data" pod "423fc4e9-6575-4690-9263-83ffff40dfc3" (UID: "423fc4e9-6575-4690-9263-83ffff40dfc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:07.565212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.565043    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423fc4e9-6575-4690-9263-83ffff40dfc3-istiod-ca-cert" pod "423fc4e9-6575-4690-9263-83ffff40dfc3" (UID: "423fc4e9-6575-4690-9263-83ffff40dfc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:07.565472 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.565409    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-certs" pod "423fc4e9-6575-4690-9263-83ffff40dfc3" (UID: "423fc4e9-6575-4690-9263-83ffff40dfc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:07.569840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.569807    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-credential-socket" pod "423fc4e9-6575-4690-9263-83ffff40dfc3" (UID: "423fc4e9-6575-4690-9263-83ffff40dfc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:07.570115 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.570086    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-envoy" pod "423fc4e9-6575-4690-9263-83ffff40dfc3" (UID: "423fc4e9-6575-4690-9263-83ffff40dfc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:07.580253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.580223    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-podinfo" pod "423fc4e9-6575-4690-9263-83ffff40dfc3" (UID: "423fc4e9-6575-4690-9263-83ffff40dfc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:09:07.584552 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.584460    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-kube-api-access-4656f" pod "423fc4e9-6575-4690-9263-83ffff40dfc3" (UID: "423fc4e9-6575-4690-9263-83ffff40dfc3"). InnerVolumeSpecName "kube-api-access-4656f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:07.584552 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.584530    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-token" pod "423fc4e9-6575-4690-9263-83ffff40dfc3" (UID: "423fc4e9-6575-4690-9263-83ffff40dfc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:07.654535 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.654497    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96964222-318a-433e-b83c-dfff82598b9a" path="/var/lib/kubelet/pods/96964222-318a-433e-b83c-dfff82598b9a/volumes"
Oct 29 22:09:07.668870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.666949    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4656f\" (UniqueName: \"kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-kube-api-access-4656f\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:07.668870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.666987    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:07.668870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.667003    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/423fc4e9-6575-4690-9263-83ffff40dfc3-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:07.668870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.667015    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:07.668870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.667026    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:07.668870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.667038    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:07.668870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.667049    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:07.668870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.667059    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/423fc4e9-6575-4690-9263-83ffff40dfc3-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:07.668870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.667071    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/423fc4e9-6575-4690-9263-83ffff40dfc3-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:07.952465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.952395    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"
Oct 29 22:09:07.952465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.952426    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8" event={"ID":"423fc4e9-6575-4690-9263-83ffff40dfc3","Type":"ContainerDied","Data":"724b7b542d9dbbc1d36181c9b33f8d96d8cda68f19bb07a2a35a9da3ce6c1377"}
Oct 29 22:09:07.952928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:07.952483    7082 scope.go:122] "RemoveContainer" containerID="0cd0fda47c37bd337f9da211613b13a3e1350a4e0aa178c0c9db90a6db351dc1"
Oct 29 22:09:08.036249 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:08.035039    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"]
Oct 29 22:09:08.043644 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:08.043593    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-734480-openshift-default-7cc69fc667-flfk8"]
Oct 29 22:09:09.637613 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:09.637578    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423fc4e9-6575-4690-9263-83ffff40dfc3" path="/var/lib/kubelet/pods/423fc4e9-6575-4690-9263-83ffff40dfc3/volumes"
Oct 29 22:09:09.649402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:09.649338    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"
Oct 29 22:09:09.649674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:09.649621    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"
Oct 29 22:09:11.084328 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:11.084287    7082 generic.go:356] "Generic (PLEG): container finished" podID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerID="3f3cf7f4c8f056359379673e4821b910c35dc16e92827754b5193f25c47f731a" exitCode=0
Oct 29 22:09:11.085523 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:11.084328    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" event={"ID":"9ef300af-2bfa-4891-9c03-e60646549fc3","Type":"ContainerDied","Data":"3f3cf7f4c8f056359379673e4821b910c35dc16e92827754b5193f25c47f731a"}
Oct 29 22:09:11.085523 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:11.084989    7082 scope.go:122] "RemoveContainer" containerID="3f3cf7f4c8f056359379673e4821b910c35dc16e92827754b5193f25c47f731a"
Oct 29 22:09:11.098009 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:11.097967    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:09:11.098271 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:11.098245    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:09:11.098431 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:11.098411    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:09:11.688776 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:11.686722    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"
Oct 29 22:09:12.107684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:12.106361    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" event={"ID":"9ef300af-2bfa-4891-9c03-e60646549fc3","Type":"ContainerStarted","Data":"a0b166882f38cf78e3417e0d5a265fb267174641c1779a4abed798b168599f1c"}
Oct 29 22:09:12.237072 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:12.236182    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"]
Oct 29 22:09:12.237072 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:12.236428    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj" podUID="35f37e68-0688-4368-9323-a03970491ce9" containerName="istio-proxy" containerID="cri-o://21d6acfe14098794b9dd99282c34622851f0d9cc1332e96c6a5ea81a65d66de9" gracePeriod=30
Oct 29 22:09:12.722408 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:12.720805    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"
Oct 29 22:09:13.022736 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:13.022249    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:09:13.022736 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:13.022379    7082 patch_prober.go:29] interesting pod/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:13.022736 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:13.022423    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 22:09:13.670335 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:13.670306    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"]
Oct 29 22:09:13.675205 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:13.675178    7082 cpu_manager.go:414] "containerMap: removing container" podUID="60fe150f-6b50-4fb2-baec-6582ba520f74" containerName="revoke-keys"
Oct 29 22:09:13.677699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:13.677661    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60fe150f-6b50-4fb2-baec-6582ba520f74" containerName="revoke-keys"
Oct 29 22:09:13.677963 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:13.677948    7082 cpu_manager.go:414] "containerMap: removing container" podUID="423fc4e9-6575-4690-9263-83ffff40dfc3" containerName="istio-proxy"
Oct 29 22:09:13.678100 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:13.678087    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423fc4e9-6575-4690-9263-83ffff40dfc3" containerName="istio-proxy"
Oct 29 22:09:13.678673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:13.678618    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="60fe150f-6b50-4fb2-baec-6582ba520f74" containerName="revoke-keys"
Oct 29 22:09:13.678804 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:13.678793    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="423fc4e9-6575-4690-9263-83ffff40dfc3" containerName="istio-proxy"
Oct 29 22:09:13.737572 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:13.737541    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:13.751344 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:13.751318    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"]
Oct 29 22:09:13.767786 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:13.767758    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b80d5b-openshift-default-dockercfg-4wczk\""
Oct 29 22:09:14.112705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.026134    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-istio-token\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.112705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.026243    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-data\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.112705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.026300    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-credential-socket\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.112705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.026417    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-certs\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.112705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.026504    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-socket\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.112705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.026550    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf767f5a-7583-4f85-a50a-70c504a28898-istiod-ca-cert\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.112705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.026660    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-envoy\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.112705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.026686    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf767f5a-7583-4f85-a50a-70c504a28898-istio-podinfo\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.112705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.026748    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn727\" (UniqueName: \"kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-kube-api-access-gn727\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.141704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.129540    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-envoy\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.141704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.129588    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf767f5a-7583-4f85-a50a-70c504a28898-istio-podinfo\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.141704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.129661    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn727\" (UniqueName: \"kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-kube-api-access-gn727\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.141704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.129798    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-istio-token\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.141704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.129878    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-data\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.141704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.129937    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-credential-socket\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.141704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.130057    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-certs\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.141704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.130144    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-socket\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.141704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.130194    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf767f5a-7583-4f85-a50a-70c504a28898-istiod-ca-cert\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.141704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.130991    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf767f5a-7583-4f85-a50a-70c504a28898-istiod-ca-cert\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.141704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.131486    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:09:14.156168 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.156139    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-data\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.156878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.156838    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-credential-socket\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.182007 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.160449    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-certs\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.182007 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.160718    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-socket\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.182007 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.171046    7082 patch_prober.go:29] interesting pod/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:14.182007 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.171132    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 22:09:14.304696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.304567    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn727\" (UniqueName: \"kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-kube-api-access-gn727\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.325083 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.325009    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-istio-token\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.325585 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.325529    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-envoy\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.343847 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.339455    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf767f5a-7583-4f85-a50a-70c504a28898-istio-podinfo\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:14.380309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:14.379946    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:15.139219 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:15.137410    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:15.184654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:15.167188    7082 patch_prober.go:29] interesting pod/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:15.184654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:15.167259    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 22:09:15.184654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:15.169596    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:15.766200 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:09:15.766159    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf767f5a_7583_4f85_a50a_70c504a28898.slice/crio-e4aa5531b14c60dabf7d9c0af4cd7dcb2353ea2b8123f4e0725edb05b80251e0 WatchSource:0}: Error finding container e4aa5531b14c60dabf7d9c0af4cd7dcb2353ea2b8123f4e0725edb05b80251e0: Status 404 returned error can't find the container with id e4aa5531b14c60dabf7d9c0af4cd7dcb2353ea2b8123f4e0725edb05b80251e0
Oct 29 22:09:15.766456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:15.766436    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"]
Oct 29 22:09:15.769360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:15.769325    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:09:15.769466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:15.769392    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:09:15.769466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:15.769423    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:09:16.022701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:16.022437    7082 patch_prober.go:29] interesting pod/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:16.022701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:16.022497    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 22:09:16.646660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:16.617730    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" event={"ID":"bf767f5a-7583-4f85-a50a-70c504a28898","Type":"ContainerStarted","Data":"a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973"}
Oct 29 22:09:16.646660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:16.617854    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" event={"ID":"bf767f5a-7583-4f85-a50a-70c504a28898","Type":"ContainerStarted","Data":"e4aa5531b14c60dabf7d9c0af4cd7dcb2353ea2b8123f4e0725edb05b80251e0"}
Oct 29 22:09:16.669685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:16.668473    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" podStartSLOduration=3.668453596 podStartE2EDuration="3.668453596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:16.666672178 +0000 UTC m=+1329.166714478" watchObservedRunningTime="2026-10-29 22:09:16.668453596 +0000 UTC m=+1329.168495896"
Oct 29 22:09:16.746302 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:16.744810    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"]
Oct 29 22:09:16.746302 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:16.745056    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn" podUID="8402b136-793a-4f5c-a73d-0a4471a9361b" containerName="payload-processing" containerID="cri-o://35755f9a900f5a07f0cdce6559c36e75f9130757f33113a67a6f548b064755b1" gracePeriod=30
Oct 29 22:09:16.750671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:16.750605    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"]
Oct 29 22:09:16.754648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:16.750819    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds" podUID="c942cc15-4baf-4eea-bd89-b9b504274bf9" containerName="payload-pre-processing" containerID="cri-o://0bb3771d468f6b5a80d49f8cb4f536b87ee5e5cb8ba8db830ed134213333ae6a" gracePeriod=30
Oct 29 22:09:17.023654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:17.022393    7082 patch_prober.go:29] interesting pod/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:17.023654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:17.022458    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 22:09:17.052449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:17.049915    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:09:17.415664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:17.403177    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:17.454663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:17.452002    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:18.259903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.259872    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"]
Oct 29 22:09:18.260331 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.259923    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d53cd-2zgsp"]
Oct 29 22:09:18.272739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.267472    7082 generic.go:356] "Generic (PLEG): container finished" podID="8402b136-793a-4f5c-a73d-0a4471a9361b" containerID="35755f9a900f5a07f0cdce6559c36e75f9130757f33113a67a6f548b064755b1" exitCode=0
Oct 29 22:09:18.272739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.267771    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn" event={"ID":"8402b136-793a-4f5c-a73d-0a4471a9361b","Type":"ContainerDied","Data":"35755f9a900f5a07f0cdce6559c36e75f9130757f33113a67a6f548b064755b1"}
Oct 29 22:09:18.272739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.268131    7082 patch_prober.go:29] interesting pod/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:18.272739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.268194    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 22:09:18.272739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.270081    7082 generic.go:356] "Generic (PLEG): container finished" podID="c942cc15-4baf-4eea-bd89-b9b504274bf9" containerID="0bb3771d468f6b5a80d49f8cb4f536b87ee5e5cb8ba8db830ed134213333ae6a" exitCode=0
Oct 29 22:09:18.272739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.271761    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds" event={"ID":"c942cc15-4baf-4eea-bd89-b9b504274bf9","Type":"ContainerDied","Data":"0bb3771d468f6b5a80d49f8cb4f536b87ee5e5cb8ba8db830ed134213333ae6a"}
Oct 29 22:09:18.272739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.271820    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:18.272739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.271852    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"]
Oct 29 22:09:18.275662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.274788    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:09:18.278423 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.277718    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b041902-qfqlp"]
Oct 29 22:09:18.284676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.283283    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"
Oct 29 22:09:18.367387 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.367343    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"
Oct 29 22:09:18.420480 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.418374    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8402b136-793a-4f5c-a73d-0a4471a9361b-kube-api-access-gk47h\" (UniqueName: \"kubernetes.io/projected/8402b136-793a-4f5c-a73d-0a4471a9361b-kube-api-access-gk47h\") pod \"8402b136-793a-4f5c-a73d-0a4471a9361b\" (UID: \"8402b136-793a-4f5c-a73d-0a4471a9361b\") "
Oct 29 22:09:18.420480 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.418820    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8402b136-793a-4f5c-a73d-0a4471a9361b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8402b136-793a-4f5c-a73d-0a4471a9361b-plugins-config-volume\") pod \"8402b136-793a-4f5c-a73d-0a4471a9361b\" (UID: \"8402b136-793a-4f5c-a73d-0a4471a9361b\") "
Oct 29 22:09:18.458997 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.449832    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8402b136-793a-4f5c-a73d-0a4471a9361b-plugins-config-volume" pod "8402b136-793a-4f5c-a73d-0a4471a9361b" (UID: "8402b136-793a-4f5c-a73d-0a4471a9361b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:18.484141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.484098    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8402b136-793a-4f5c-a73d-0a4471a9361b-kube-api-access-gk47h" pod "8402b136-793a-4f5c-a73d-0a4471a9361b" (UID: "8402b136-793a-4f5c-a73d-0a4471a9361b"). InnerVolumeSpecName "kube-api-access-gk47h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:18.528735 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.521134    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c942cc15-4baf-4eea-bd89-b9b504274bf9-kube-api-access-8fjcs\" (UniqueName: \"kubernetes.io/projected/c942cc15-4baf-4eea-bd89-b9b504274bf9-kube-api-access-8fjcs\") pod \"c942cc15-4baf-4eea-bd89-b9b504274bf9\" (UID: \"c942cc15-4baf-4eea-bd89-b9b504274bf9\") "
Oct 29 22:09:18.528735 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.521351    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c942cc15-4baf-4eea-bd89-b9b504274bf9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c942cc15-4baf-4eea-bd89-b9b504274bf9-plugins-config-volume\") pod \"c942cc15-4baf-4eea-bd89-b9b504274bf9\" (UID: \"c942cc15-4baf-4eea-bd89-b9b504274bf9\") "
Oct 29 22:09:18.528735 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.526308    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c942cc15-4baf-4eea-bd89-b9b504274bf9-plugins-config-volume" pod "c942cc15-4baf-4eea-bd89-b9b504274bf9" (UID: "c942cc15-4baf-4eea-bd89-b9b504274bf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:18.528735 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.526699    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8402b136-793a-4f5c-a73d-0a4471a9361b-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:18.528735 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.526725    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c942cc15-4baf-4eea-bd89-b9b504274bf9-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:18.528735 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.526739    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk47h\" (UniqueName: \"kubernetes.io/projected/8402b136-793a-4f5c-a73d-0a4471a9361b-kube-api-access-gk47h\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:18.545036 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.544999    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c942cc15-4baf-4eea-bd89-b9b504274bf9-kube-api-access-8fjcs" pod "c942cc15-4baf-4eea-bd89-b9b504274bf9" (UID: "c942cc15-4baf-4eea-bd89-b9b504274bf9"). InnerVolumeSpecName "kube-api-access-8fjcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:18.563834 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.563772    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"]
Oct 29 22:09:18.568147 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.568013    7082 cpu_manager.go:414] "containerMap: removing container" podUID="423fc4e9-6575-4690-9263-83ffff40dfc3" containerName="istio-proxy"
Oct 29 22:09:18.568147 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.568048    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423fc4e9-6575-4690-9263-83ffff40dfc3" containerName="istio-proxy"
Oct 29 22:09:18.568147 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.568128    7082 cpu_manager.go:414] "containerMap: removing container" podUID="c942cc15-4baf-4eea-bd89-b9b504274bf9" containerName="payload-pre-processing"
Oct 29 22:09:18.568147 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.568139    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c942cc15-4baf-4eea-bd89-b9b504274bf9" containerName="payload-pre-processing"
Oct 29 22:09:18.568579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.568202    7082 cpu_manager.go:414] "containerMap: removing container" podUID="8402b136-793a-4f5c-a73d-0a4471a9361b" containerName="payload-processing"
Oct 29 22:09:18.568579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.568213    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8402b136-793a-4f5c-a73d-0a4471a9361b" containerName="payload-processing"
Oct 29 22:09:18.568579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.568413    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="c942cc15-4baf-4eea-bd89-b9b504274bf9" containerName="payload-pre-processing"
Oct 29 22:09:18.568579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.568452    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="8402b136-793a-4f5c-a73d-0a4471a9361b" containerName="payload-processing"
Oct 29 22:09:18.568579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.568470    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="423fc4e9-6575-4690-9263-83ffff40dfc3" containerName="istio-proxy"
Oct 29 22:09:18.591598 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.590822    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"]
Oct 29 22:09:18.591598 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.590957    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"
Oct 29 22:09:18.608004 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.607796    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:09:18.608004 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.607895    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:09:18.608004 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.607944    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:09:18.608004 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.607978    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:09:18.637543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.636730    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fjcs\" (UniqueName: \"kubernetes.io/projected/c942cc15-4baf-4eea-bd89-b9b504274bf9-kube-api-access-8fjcs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:18.738579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.738536    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5vvt\" (UniqueName: \"kubernetes.io/projected/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-kube-api-access-z5vvt\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg\" (UID: \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"
Oct 29 22:09:18.738774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.738703    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg\" (UID: \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"
Oct 29 22:09:18.738825 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.738804    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg\" (UID: \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"
Oct 29 22:09:18.841235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.841196    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg\" (UID: \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"
Oct 29 22:09:18.841827 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.841802    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5vvt\" (UniqueName: \"kubernetes.io/projected/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-kube-api-access-z5vvt\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg\" (UID: \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"
Oct 29 22:09:18.842100 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.842080    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg\" (UID: \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"
Oct 29 22:09:18.842864 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.842837    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg\" (UID: \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"
Oct 29 22:09:18.844059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.844027    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg\" (UID: \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"
Oct 29 22:09:18.880360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.880323    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds" podUID="c942cc15-4baf-4eea-bd89-b9b504274bf9" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.166:9004: i/o timeout"
Oct 29 22:09:18.916704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.914397    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5vvt\" (UniqueName: \"kubernetes.io/projected/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-kube-api-access-z5vvt\") pod \"maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg\" (UID: \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"
Oct 29 22:09:18.961651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:18.959399    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"
Oct 29 22:09:19.026978 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.026928    7082 patch_prober.go:29] interesting pod/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:19.027268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.027197    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 22:09:19.348181 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.348065    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"
Oct 29 22:09:19.348181 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.348124    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn" event={"ID":"8402b136-793a-4f5c-a73d-0a4471a9361b","Type":"ContainerDied","Data":"8516ea39e10546ea8fafada5cbb1e298972e37dccf7537b11e90c4f77ce37b60"}
Oct 29 22:09:19.348874 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.348197    7082 scope.go:122] "RemoveContainer" containerID="35755f9a900f5a07f0cdce6559c36e75f9130757f33113a67a6f548b064755b1"
Oct 29 22:09:19.360723 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.357136    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"
Oct 29 22:09:19.360723 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.359587    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds" event={"ID":"c942cc15-4baf-4eea-bd89-b9b504274bf9","Type":"ContainerDied","Data":"19574baab2f65631c5186c428e21dad63ec9e6b96a009abda120c2ac1dd017b0"}
Oct 29 22:09:19.360723 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.359652    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"]
Oct 29 22:09:19.362306 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:09:19.362286    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8276481_b6a7_40d9_b2f1_99c1d8de9bb1.slice/crio-0d1c7248586ce75cdf95b27671bc3574cc18c46c9f72367f13e2f51f5b724821 WatchSource:0}: Error finding container 0d1c7248586ce75cdf95b27671bc3574cc18c46c9f72367f13e2f51f5b724821: Status 404 returned error can't find the container with id 0d1c7248586ce75cdf95b27671bc3574cc18c46c9f72367f13e2f51f5b724821
Oct 29 22:09:19.375619 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.373308    7082 scope.go:122] "RemoveContainer" containerID="0bb3771d468f6b5a80d49f8cb4f536b87ee5e5cb8ba8db830ed134213333ae6a"
Oct 29 22:09:19.440151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.440029    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"]
Oct 29 22:09:19.470291 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.470256    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d53cd-5d999bd97d-lk2qn"]
Oct 29 22:09:19.495486 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.495454    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"]
Oct 29 22:09:19.503247 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.503200    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-5876c5456c-m8mds"]
Oct 29 22:09:19.796735 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.791688    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c3563f4-8c85-4267-acf0-020554ae7aa2" path="/var/lib/kubelet/pods/1c3563f4-8c85-4267-acf0-020554ae7aa2/volumes"
Oct 29 22:09:19.799072 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.798390    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60fe150f-6b50-4fb2-baec-6582ba520f74" path="/var/lib/kubelet/pods/60fe150f-6b50-4fb2-baec-6582ba520f74/volumes"
Oct 29 22:09:19.799231 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.799147    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8402b136-793a-4f5c-a73d-0a4471a9361b" path="/var/lib/kubelet/pods/8402b136-793a-4f5c-a73d-0a4471a9361b/volumes"
Oct 29 22:09:19.819917 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:19.819759    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c942cc15-4baf-4eea-bd89-b9b504274bf9" path="/var/lib/kubelet/pods/c942cc15-4baf-4eea-bd89-b9b504274bf9/volumes"
Oct 29 22:09:20.029505 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:20.029461    7082 patch_prober.go:29] interesting pod/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:20.029794 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:20.029757    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 29 22:09:20.503668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:20.503563    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg" event={"ID":"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1","Type":"ContainerStarted","Data":"0d1c7248586ce75cdf95b27671bc3574cc18c46c9f72367f13e2f51f5b724821"}
Oct 29 22:09:21.171648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:21.169022    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:09:21.195649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:21.189556    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:09:21.563866 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:21.563771    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg" event={"ID":"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1","Type":"ContainerStarted","Data":"c0a70ba57efacfd383da535aa1b018151b9b24834fb0be94e840efb154fc22e5"}
Oct 29 22:09:21.725704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:21.714227    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg" podStartSLOduration=3.714207565 podStartE2EDuration="3.714207565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:21.617980741 +0000 UTC m=+1334.118023041" watchObservedRunningTime="2026-10-29 22:09:21.714207565 +0000 UTC m=+1334.214249866"
Oct 29 22:09:22.688897 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:22.688858    7082 generic.go:356] "Generic (PLEG): container finished" podID="a8276481-b6a7-40d9-b2f1-99c1d8de9bb1" containerID="c0a70ba57efacfd383da535aa1b018151b9b24834fb0be94e840efb154fc22e5" exitCode=0
Oct 29 22:09:22.700649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:22.689606    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg" event={"ID":"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1","Type":"ContainerDied","Data":"c0a70ba57efacfd383da535aa1b018151b9b24834fb0be94e840efb154fc22e5"}
Oct 29 22:09:22.701557 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:22.701525    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:09:22.726722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:22.726675    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:22.770655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:22.752265    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:22.771095 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:22.770966    7082 patch_prober.go:29] interesting pod/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:22.771095 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:22.771028    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj" podUID="35f37e68-0688-4368-9323-a03970491ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:22.820815 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:22.820742    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:22.824079 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:22.821476    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:22.824079 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:22.821512    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:09:23.566655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:23.546598    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Oct 29 22:09:23.708022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:23.707561    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:09:23.710701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:23.708750    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:09:23.730818 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:23.730718    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.778481 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:23.764577    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:09:24.754656 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:09:24.732375    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.022s"
Oct 29 22:09:24.855355 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:24.855325    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:09:24.859840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:24.859802    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:09:24.906550 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:24.906513    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.006866 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:25.006772    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:09:25.042199 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:25.042155    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:09:25.046406 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:25.045297    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:09:25.130858 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:25.109913    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:25.131828 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:25.131789    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.207700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:25.176339    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:09:25.893128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:25.223303    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:09:25.893128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:25.360165    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": context deadline exceeded"
Oct 29 22:09:25.893128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:25.496688    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:09:25.982073 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:25.947881    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:09:26.067770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:26.067720    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:26.138655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:26.091094    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:09:26.722651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:26.721832    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:26.841668 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:09:26.812761    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.131s"
Oct 29 22:09:26.841668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:26.812863    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:09:26.841668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:26.812969    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"]
Oct 29 22:09:26.905082 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:26.885717    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:09:26.989054 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:26.892343    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"]
Oct 29 22:09:26.989054 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:26.956880    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:09:26.989851 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:26.989801    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.000857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.000815    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.001093 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.001072    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.001209 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.001195    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.001392 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.001373    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.001646 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.001610    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.282080 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.015244    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:27.300131 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:26.958179    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.313942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:26.956915    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.328644 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.172769    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:09:27.346361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.249124    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.380657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.378529    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-48c52b-openshift-default-dockercfg-zsvgm\""
Oct 29 22:09:27.399683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.296251    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.399683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.387759    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79nx\" (UniqueName: \"kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-kube-api-access-d79nx\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.399683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.269431    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.399683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.387936    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.505836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.497202    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.505836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.497410    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.505836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.497693    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.505836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.497727    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d79nx\" (UniqueName: \"kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-kube-api-access-d79nx\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.505836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.497833    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.505836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.497986    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.505836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.498095    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.505836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.498137    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.505836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.498164    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.505836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.498778    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.508337 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.507819    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.509385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.508480    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.513649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.509868    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.516562 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.516536    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.711654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.571399    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.711654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.572046    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79nx\" (UniqueName: \"kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-kube-api-access-d79nx\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.711654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.572889    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:27.725777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.725738    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.925066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:27.924999    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:28.019726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:28.007077    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.027919 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:28.027863    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:09:28.046859 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:28.046484    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:28.108032 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:28.088575    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:28.108032 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:28.088648    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.109941 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:28.109894    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:28.121653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:28.117723    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.314657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:28.309326    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.424841 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:28.424796    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:09:28.491955 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:28.330467    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:29.158814 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:29.158777    7082 patch_prober.go:29] interesting pod/console-5bd87bd5df-9rfzx container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.20:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:29.159416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:29.159381    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-5bd87bd5df-9rfzx" podUID="4a9db242-c903-49e3-92a6-fdf982c8f25e" containerName="console" probeResult="failure" output="Get \"https://10.134.0.20:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.159583 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:28.547333    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.195555 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:29.195511    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:29.210887 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:29.210844    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs" podUID="c32b079b-eba6-4e6f-8f79-f9c3cb86de99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:09:29.445708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:29.425977    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:09:29.445708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:29.426505    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:09:29.445708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:28.491898    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.521721 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:29.490741    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.794667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:29.794534    7082 scope.go:122] "RemoveContainer" containerID="49f86a0498b29ba75b734a43a50a0c3244e9389a11448f3bd1b17cc72a842235"
Oct 29 22:09:29.813063 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:29.795034    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" podUID="0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.895688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:29.859727    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:09:29.895688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:29.891995    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded"
Oct 29 22:09:29.938978 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:29.936221    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.078716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.074787    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:30.079001 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.078968    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.264151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.264103    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:30.427731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.426726    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:09:30.452652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.449689    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.529860 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.526909    7082 scope.go:122] "RemoveContainer" containerID="fe9b4c9b13beaac0f41c209b78d27f8ca5b40b1578333f18131e6fc5e6e96eab"
Oct 29 22:09:30.529860 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.529343    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.531134 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.531105    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded"
Oct 29 22:09:30.531445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.531425    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.591649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.550122    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:09:30.611326 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:09:30.610587    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:09:30.676821 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.676764    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:30.677103 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.677065    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.737853 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.727291    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"]
Oct 29 22:09:30.869882 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.864736    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.898056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.898014    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded"
Oct 29 22:09:30.926590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.899410    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"
Oct 29 22:09:30.926590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.899497    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"
Oct 29 22:09:30.926590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.916946    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b80d5b\""
Oct 29 22:09:30.926590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.917484    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b80d5b-dockercfg-mtcrq\""
Oct 29 22:09:30.947925 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.947698    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"]
Oct 29 22:09:30.955177 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.952781    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:30.955177 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:30.952852    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.109660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.079342    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-xks-ca\") pod \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\" (UID: \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\") "
Oct 29 22:09:31.109660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.079408    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-openshift-ca\") pod \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\" (UID: \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\") "
Oct 29 22:09:31.109660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.079659    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-kube-api-access-z5vvt\" (UniqueName: \"kubernetes.io/projected/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-kube-api-access-z5vvt\") pod \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\" (UID: \"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1\") "
Oct 29 22:09:31.109660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.080116    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65ddf70-7369-430f-bb7d-ac3e2786f92d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng\" (UID: \"f65ddf70-7369-430f-bb7d-ac3e2786f92d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"
Oct 29 22:09:31.109660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.080190    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968sn\" (UniqueName: \"kubernetes.io/projected/f65ddf70-7369-430f-bb7d-ac3e2786f92d-kube-api-access-968sn\") pod \"payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng\" (UID: \"f65ddf70-7369-430f-bb7d-ac3e2786f92d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"
Oct 29 22:09:31.185690 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.184882    7082 scope.go:122] "RemoveContainer" containerID="2e9291d2ae8808796ad29701fa4e2966804d433c272a19f51aaa05ddc2a5f231"
Oct 29 22:09:31.517246 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.517207    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-openshift-ca" pod "a8276481-b6a7-40d9-b2f1-99c1d8de9bb1" (UID: "a8276481-b6a7-40d9-b2f1-99c1d8de9bb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:31.544682 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.538172    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65ddf70-7369-430f-bb7d-ac3e2786f92d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng\" (UID: \"f65ddf70-7369-430f-bb7d-ac3e2786f92d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"
Oct 29 22:09:31.544682 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.538269    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-968sn\" (UniqueName: \"kubernetes.io/projected/f65ddf70-7369-430f-bb7d-ac3e2786f92d-kube-api-access-968sn\") pod \"payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng\" (UID: \"f65ddf70-7369-430f-bb7d-ac3e2786f92d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"
Oct 29 22:09:31.544682 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.538769    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:31.544682 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.539990    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65ddf70-7369-430f-bb7d-ac3e2786f92d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng\" (UID: \"f65ddf70-7369-430f-bb7d-ac3e2786f92d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"
Oct 29 22:09:31.548665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.546383    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.570448 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.550767    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:31.570448 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.551054    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:09:31.574669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.574004    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:09:31.594747 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.594709    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.620537 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.620494    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.893851 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.879702    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-968sn\" (UniqueName: \"kubernetes.io/projected/f65ddf70-7369-430f-bb7d-ac3e2786f92d-kube-api-access-968sn\") pod \"payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng\" (UID: \"f65ddf70-7369-430f-bb7d-ac3e2786f92d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"
Oct 29 22:09:31.893851 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.880031    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.959252 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.957885    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg" event={"ID":"a8276481-b6a7-40d9-b2f1-99c1d8de9bb1","Type":"ContainerDied","Data":"0d1c7248586ce75cdf95b27671bc3574cc18c46c9f72367f13e2f51f5b724821"}
Oct 29 22:09:31.959252 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.957954    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d1c7248586ce75cdf95b27671bc3574cc18c46c9f72367f13e2f51f5b724821"
Oct 29 22:09:31.959252 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.958031    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"
Oct 29 22:09:31.959840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.959797    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-xks-ca" pod "a8276481-b6a7-40d9-b2f1-99c1d8de9bb1" (UID: "a8276481-b6a7-40d9-b2f1-99c1d8de9bb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:09:31.967875 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.967825    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"]
Oct 29 22:09:31.979129 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.979098    7082 cpu_manager.go:414] "containerMap: removing container" podUID="a8276481-b6a7-40d9-b2f1-99c1d8de9bb1" containerName="revoke-keys"
Oct 29 22:09:31.979372 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.979353    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8276481-b6a7-40d9-b2f1-99c1d8de9bb1" containerName="revoke-keys"
Oct 29 22:09:31.988006 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:31.987967    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8276481-b6a7-40d9-b2f1-99c1d8de9bb1" containerName="revoke-keys"
Oct 29 22:09:32.007747 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.006782    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:32.007747 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.006839    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.007747 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.007478    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"
Oct 29 22:09:32.081073 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.081036    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:32.146779 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.146679    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"]
Oct 29 22:09:32.561954 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.504467    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-27cda7-openshift-default-dockercfg-hn67d\""
Oct 29 22:09:32.673727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.611813    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:09:32.673727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.660588    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.673727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.661646    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:32.673727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.661884    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:09:32.673727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.662098    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:09:32.673727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.662342    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:09:32.760068 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.678916    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.937713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.894401    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.937713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:32.924185    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.029172    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-socket\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.029275    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k8fz\" (UniqueName: \"kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-kube-api-access-2k8fz\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.029378    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-credential-socket\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.029415    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istiod-ca-cert\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.029446    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-envoy\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.029682    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-token\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.029753    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-certs\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.029815    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-data\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.029852    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-podinfo\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.096713    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"]
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.137388    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-token\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.137495    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-certs\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.137557    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-data\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.137593    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-podinfo\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.139515    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-certs\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.139698    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-kube-api-access-z5vvt" pod "a8276481-b6a7-40d9-b2f1-99c1d8de9bb1" (UID: "a8276481-b6a7-40d9-b2f1-99c1d8de9bb1"). InnerVolumeSpecName "kube-api-access-z5vvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:33.162655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.140565    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-data\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.189888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.188734    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-socket\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.189888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.188884    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k8fz\" (UniqueName: \"kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-kube-api-access-2k8fz\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.189888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.189013    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-credential-socket\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.189888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.189058    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istiod-ca-cert\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.189888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.189096    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-envoy\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.189888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.189393    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5vvt\" (UniqueName: \"kubernetes.io/projected/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1-kube-api-access-z5vvt\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:33.221215 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.202289    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-credential-socket\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.221215 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.221160    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istiod-ca-cert\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.343149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.334311    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-socket\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.598386 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.495034    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k8fz\" (UniqueName: \"kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-kube-api-access-2k8fz\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.629649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.608446    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:09:33.629649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.608549    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:09:33.629649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.608597    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:09:33.629649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.609328    7082 scope.go:122] "RemoveContainer" containerID="19d07e36c3b8e9325d2ae74625023a21c6cbcb93cf7f3a8a6d12ed170a753572"
Oct 29 22:09:33.659666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.639724    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.799830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.669152    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.799830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.671135    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.799830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.673289    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.799830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.676749    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.799830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.757468    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.799830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.772234    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.825454 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.822255    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-podinfo\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.829105 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.829017    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-envoy\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:33.925707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:33.912829    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.313709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.302924    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08f293db-f5865"]
Oct 29 22:09:34.452311 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.452274    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-token\") pod \"e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") " pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:34.548691 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.539744    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:34.607735 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.575169    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:34.607735 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.575245    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.662322 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.628174    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:34.662322 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.628238    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.662322 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.652020    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.691202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.668751    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:34.691202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.668829    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.691202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.671711    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.691202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.672098    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.691202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.674143    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.734650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.691737    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.734650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.722314    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.757953 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.757910    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:09:34.840465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.832097    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" event={"ID":"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6","Type":"ContainerStarted","Data":"657ce4a2b69cc9f32498cd6d11a4c65745b96075788cd21628db2e66c860e168"}
Oct 29 22:09:34.840465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.832903    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:34.840465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.832958    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.950768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:34.922764    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.378109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:35.309739    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": context deadline exceeded"
Oct 29 22:09:35.767231 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:35.686311    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.767231 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:35.737863    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.819994 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:35.819891    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.138899 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:36.038298    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.441392 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:35.829803    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:09:36.455785 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:35.829735    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.455785 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:36.451698    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs" podUID="c32b079b-eba6-4e6f-8f79-f9c3cb86de99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.455785 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:36.359224    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.873660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:36.868590    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:37.004684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:36.974108    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" podUID="4684c6fd-8ac7-4005-a20d-48396c0ab981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:37.004684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:36.984393    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:37.022799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:37.022457    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:09:37.321858 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:37.283048    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5" path="/var/lib/kubelet/pods/a1bd3607-1cfd-4d2d-b67e-1c0b933dc7e5/volumes"
Oct 29 22:09:37.406802 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:09:37.381339    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.733s"
Oct 29 22:09:37.407086 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:37.407059    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"]
Oct 29 22:09:37.424846 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:37.407376    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"]
Oct 29 22:09:37.448128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:37.430435    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:37.519655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:37.432975    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:37.961841 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:37.961736    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:09:37.998658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:37.993029    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:38.076066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:38.076003    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:38.279711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:38.245761    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:09:38.282787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:38.282744    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:38.283829 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:38.283787    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:09:38.607932 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:38.607866    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:09:38.809041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:38.621784    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:09:38.809376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:38.729190    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" podUID="4684c6fd-8ac7-4005-a20d-48396c0ab981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:09:38.809559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:38.756834    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt" podUID="f069baec-59d9-4c9e-b5a2-c25261d01cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:38.809725 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:38.808801    7082 patch_prober.go:29] interesting pod/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:38.810075 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:38.810035    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj" podUID="35f37e68-0688-4368-9323-a03970491ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:39.508704 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:09:39.508657    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.487s"
Oct 29 22:09:39.513918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:39.509335    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 22:09:40.218141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:40.218085    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:09:40.235902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:40.235871    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:09:40.236772 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:40.216807    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:40.278571 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:40.237134    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:09:40.279308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:40.235804    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:40.314539 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:40.314501    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:40.374167 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:40.374123    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:09:40.389655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:40.374888    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:40.438698 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:40.425655    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:40.978339 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:09:40.975290    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.344s"
Oct 29 22:09:40.978339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:40.975375    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx"
Oct 29 22:09:41.218845 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.218794    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:41.244106 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.244062    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:41.286780 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.286747    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"]
Oct 29 22:09:41.293871 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.289002    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" event={"ID":"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6","Type":"ContainerStarted","Data":"3d32fa4d317e07e36677770998a56d106e1161ca3a787fb645f99f769272193e"}
Oct 29 22:09:41.328650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.321607    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:41.328650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.322690    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:41.328650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.323212    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:09:41.408653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.408596    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:09:41.458319 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.458281    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng" event={"ID":"f65ddf70-7369-430f-bb7d-ac3e2786f92d","Type":"ContainerStarted","Data":"12548a1c86aec217181825163c30c0567d5feca5a175312e4f98a835bed92e94"}
Oct 29 22:09:41.481162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.481102    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:09:41.543435 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.543373    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:09:41.543746 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.543697    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:09:41.543952 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.543914    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:09:41.695730 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:41.695653    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" podStartSLOduration=18.695604155 podStartE2EDuration="18.695604155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:41.502719354 +0000 UTC m=+1354.002761667" watchObservedRunningTime="2026-10-29 22:09:41.695604155 +0000 UTC m=+1354.195646462"
Oct 29 22:09:42.150296 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:42.150249    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:42.278990 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:42.254190    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:42.278990 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:42.229960    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:09:42.278990 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:42.181358    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:42.278990 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:42.277771    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:09:42.349033 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:42.327818    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:42.349033 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:42.328723    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:09:42.435719 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:42.411513    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:42.494650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:42.479398    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:09:42.825149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:42.820841    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:09:43.085775 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:43.043777    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng" event={"ID":"f65ddf70-7369-430f-bb7d-ac3e2786f92d","Type":"ContainerStarted","Data":"44c48537cfff6ac9d6d0bc036f399f5ec9242599ef22e11acb3c34814b9c0b54"}
Oct 29 22:09:43.085775 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:43.045648    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"
Oct 29 22:09:43.158449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:43.134706    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:43.158449 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:43.150064    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:09:43.199742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:43.199691    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" event={"ID":"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248","Type":"ContainerStarted","Data":"20a39401edad6ecf416f8188fcaa236d7d185b1c98b2e9d55ba8b102b5224a33"}
Oct 29 22:09:43.331665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:43.329312    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:43.331665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:43.330539    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:09:43.416709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:43.415892    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:43.455672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:43.446414    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:43.510805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:43.510738    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:09:43.572981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:43.558139    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:09:43.822536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:43.822494    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:43.937539 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:43.937006    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Oct 29 22:09:44.331378 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:44.331334    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:44.332660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:44.332323    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:09:44.454183 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:44.454145    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:09:44.455044 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:44.454977    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng" podStartSLOduration=18.454958212 podStartE2EDuration="18.454958212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:43.115498024 +0000 UTC m=+1355.615540325" watchObservedRunningTime="2026-10-29 22:09:44.454958212 +0000 UTC m=+1356.955000509"
Oct 29 22:09:44.972234 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:44.971531    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:09:45.060905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.032212    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": context deadline exceeded"
Oct 29 22:09:45.060905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.032947    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:45.060905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.052893    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:09:45.060905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.058540    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:09:45.067955 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.067919    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:09:45.076657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.075752    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:45.359194 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.359138    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:09:45.394658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.394606    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:09:45.455698 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.455146    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:09:45.493722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.493691    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:09:45.494439 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.494415    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:09:45.504727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.496210    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:09:45.505566 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.505449    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:09:45.587863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.587173    7082 generic.go:356] "Generic (PLEG): container finished" podID="35f37e68-0688-4368-9323-a03970491ce9" containerID="21d6acfe14098794b9dd99282c34622851f0d9cc1332e96c6a5ea81a65d66de9" exitCode=137
Oct 29 22:09:45.587863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.587555    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj" event={"ID":"35f37e68-0688-4368-9323-a03970491ce9","Type":"ContainerDied","Data":"21d6acfe14098794b9dd99282c34622851f0d9cc1332e96c6a5ea81a65d66de9"}
Oct 29 22:09:45.598148 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.598093    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:09:45.604348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.604250    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" event={"ID":"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248","Type":"ContainerStarted","Data":"5e5971e8c0ee066cbbaa007def83ced34ee096d0d8d2375740f7302916d71340"}
Oct 29 22:09:45.638655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.610786    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:09:45.650060 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.650013    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:09:45.670673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.670491    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:09:45.741600 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.741355    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-data\") pod \"35f37e68-0688-4368-9323-a03970491ce9\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") "
Oct 29 22:09:45.741984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.741572    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-socket\") pod \"35f37e68-0688-4368-9323-a03970491ce9\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") "
Oct 29 22:09:45.741984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.741883    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-credential-socket\") pod \"35f37e68-0688-4368-9323-a03970491ce9\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") "
Oct 29 22:09:45.742656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.742207    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35f37e68-0688-4368-9323-a03970491ce9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f37e68-0688-4368-9323-a03970491ce9-istio-podinfo\") pod \"35f37e68-0688-4368-9323-a03970491ce9\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") "
Oct 29 22:09:45.742656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.742364    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-envoy\") pod \"35f37e68-0688-4368-9323-a03970491ce9\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") "
Oct 29 22:09:45.742656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.742402    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-certs\") pod \"35f37e68-0688-4368-9323-a03970491ce9\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") "
Oct 29 22:09:45.742656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.742504    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-istio-token\" (UniqueName: \"kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-istio-token\") pod \"35f37e68-0688-4368-9323-a03970491ce9\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") "
Oct 29 22:09:45.742656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.742530    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f37e68-0688-4368-9323-a03970491ce9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f37e68-0688-4368-9323-a03970491ce9-istiod-ca-cert\") pod \"35f37e68-0688-4368-9323-a03970491ce9\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") "
Oct 29 22:09:45.742656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.742580    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-kube-api-access-xqlbm\" (UniqueName: \"kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-kube-api-access-xqlbm\") pod \"35f37e68-0688-4368-9323-a03970491ce9\" (UID: \"35f37e68-0688-4368-9323-a03970491ce9\") "
Oct 29 22:09:45.745279 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.745092    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-data" pod "35f37e68-0688-4368-9323-a03970491ce9" (UID: "35f37e68-0688-4368-9323-a03970491ce9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:45.745685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.745446    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-socket" pod "35f37e68-0688-4368-9323-a03970491ce9" (UID: "35f37e68-0688-4368-9323-a03970491ce9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:45.746115 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.745826    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-credential-socket" pod "35f37e68-0688-4368-9323-a03970491ce9" (UID: "35f37e68-0688-4368-9323-a03970491ce9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:45.752107 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.751925    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-envoy" pod "35f37e68-0688-4368-9323-a03970491ce9" (UID: "35f37e68-0688-4368-9323-a03970491ce9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:45.753287 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.752304    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-certs" pod "35f37e68-0688-4368-9323-a03970491ce9" (UID: "35f37e68-0688-4368-9323-a03970491ce9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:09:45.758813 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.758750    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-istio-token" pod "35f37e68-0688-4368-9323-a03970491ce9" (UID: "35f37e68-0688-4368-9323-a03970491ce9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:45.759666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.759370    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f37e68-0688-4368-9323-a03970491ce9-istiod-ca-cert" pod "35f37e68-0688-4368-9323-a03970491ce9" (UID: "35f37e68-0688-4368-9323-a03970491ce9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:45.768755 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.766045    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-kube-api-access-xqlbm" pod "35f37e68-0688-4368-9323-a03970491ce9" (UID: "35f37e68-0688-4368-9323-a03970491ce9"). InnerVolumeSpecName "kube-api-access-xqlbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:45.768755 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.766749    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35f37e68-0688-4368-9323-a03970491ce9-istio-podinfo" pod "35f37e68-0688-4368-9323-a03970491ce9" (UID: "35f37e68-0688-4368-9323-a03970491ce9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:09:45.849265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.847272    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:45.849265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.847307    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f37e68-0688-4368-9323-a03970491ce9-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:45.849265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.847323    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqlbm\" (UniqueName: \"kubernetes.io/projected/35f37e68-0688-4368-9323-a03970491ce9-kube-api-access-xqlbm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:45.849265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.847340    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:45.849265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.847355    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:45.849265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.847369    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:45.849265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.847384    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f37e68-0688-4368-9323-a03970491ce9-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:45.849265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.847398    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:45.849265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.847412    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f37e68-0688-4368-9323-a03970491ce9-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:45.854863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.850914    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"]
Oct 29 22:09:45.854863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.852240    7082 cpu_manager.go:414] "containerMap: removing container" podUID="35f37e68-0688-4368-9323-a03970491ce9" containerName="istio-proxy"
Oct 29 22:09:45.854863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.852263    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f37e68-0688-4368-9323-a03970491ce9" containerName="istio-proxy"
Oct 29 22:09:45.854863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.852543    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f37e68-0688-4368-9323-a03970491ce9" containerName="istio-proxy"
Oct 29 22:09:45.859358 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.858640    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"
Oct 29 22:09:45.878052 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.878019    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"]
Oct 29 22:09:45.882978 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.882940    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-48c52b\""
Oct 29 22:09:45.900573 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.900541    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-48c52b-dockercfg-7njvd\""
Oct 29 22:09:45.949721 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.949673    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvkdc\" (UniqueName: \"kubernetes.io/projected/7931685e-46cf-4f26-9af5-d0e43e44997d-kube-api-access-xvkdc\") pod \"payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b\" (UID: \"7931685e-46cf-4f26-9af5-d0e43e44997d\") " pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"
Oct 29 22:09:45.949903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:45.949766    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7931685e-46cf-4f26-9af5-d0e43e44997d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b\" (UID: \"7931685e-46cf-4f26-9af5-d0e43e44997d\") " pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"
Oct 29 22:09:46.061478 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.061437    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkdc\" (UniqueName: \"kubernetes.io/projected/7931685e-46cf-4f26-9af5-d0e43e44997d-kube-api-access-xvkdc\") pod \"payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b\" (UID: \"7931685e-46cf-4f26-9af5-d0e43e44997d\") " pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"
Oct 29 22:09:46.061767 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.061542    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7931685e-46cf-4f26-9af5-d0e43e44997d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b\" (UID: \"7931685e-46cf-4f26-9af5-d0e43e44997d\") " pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"
Oct 29 22:09:46.064194 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.062855    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7931685e-46cf-4f26-9af5-d0e43e44997d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b\" (UID: \"7931685e-46cf-4f26-9af5-d0e43e44997d\") " pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"
Oct 29 22:09:46.066028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.065983    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" podStartSLOduration=18.065965889 podStartE2EDuration="18.065965889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:46.030215879 +0000 UTC m=+1358.530258179" watchObservedRunningTime="2026-10-29 22:09:46.065965889 +0000 UTC m=+1358.566008186"
Oct 29 22:09:46.093737 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.093618    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkdc\" (UniqueName: \"kubernetes.io/projected/7931685e-46cf-4f26-9af5-d0e43e44997d-kube-api-access-xvkdc\") pod \"payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b\" (UID: \"7931685e-46cf-4f26-9af5-d0e43e44997d\") " pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"
Oct 29 22:09:46.225700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.217028    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"
Oct 29 22:09:46.587557 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.586447    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:46.617219 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.611615    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:46.916440 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.915979    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj" event={"ID":"35f37e68-0688-4368-9323-a03970491ce9","Type":"ContainerDied","Data":"47de5575b1381c9bc2e434b4f237e84ed64380af4290bbb764eb344accad1fde"}
Oct 29 22:09:46.916440 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.916110    7082 scope.go:122] "RemoveContainer" containerID="21d6acfe14098794b9dd99282c34622851f0d9cc1332e96c6a5ea81a65d66de9"
Oct 29 22:09:46.916440 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.916335    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"
Oct 29 22:09:46.926324 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.926249    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:46.928396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.928354    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:09:46.937847 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:46.937818    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"
Oct 29 22:09:47.151331 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:47.151239    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"]
Oct 29 22:09:47.269456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:47.269289    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a26f29-openshift-default-f5d6bf475-2k2cj"]
Oct 29 22:09:47.305662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:47.305617    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"]
Oct 29 22:09:47.643113 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:47.643070    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f37e68-0688-4368-9323-a03970491ce9" path="/var/lib/kubelet/pods/35f37e68-0688-4368-9323-a03970491ce9/volumes"
Oct 29 22:09:47.945046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:47.943747    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b" event={"ID":"7931685e-46cf-4f26-9af5-d0e43e44997d","Type":"ContainerStarted","Data":"25401049a108a54217a886fd6522915403252c84d886bf5736a4501a115e9a25"}
Oct 29 22:09:47.945046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:47.943819    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b" event={"ID":"7931685e-46cf-4f26-9af5-d0e43e44997d","Type":"ContainerStarted","Data":"744b97b1c05d380c819dd69d5c703c6e476eef6a9c3e594ce7b6cabb48d1d14f"}
Oct 29 22:09:47.945046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:47.944736    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"
Oct 29 22:09:47.996245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:47.996161    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b" podStartSLOduration=2.996140796 podStartE2EDuration="2.996140796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:47.988234694 +0000 UTC m=+1360.488276993" watchObservedRunningTime="2026-10-29 22:09:47.996140796 +0000 UTC m=+1360.496183096"
Oct 29 22:09:48.031829 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:48.031788    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"]
Oct 29 22:09:48.032450 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:48.032058    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs" podUID="08aebda0-8d4c-433a-b844-2845c9a35971" containerName="payload-processing" containerID="cri-o://30728a04fe0467c1efa51df7b1a1dd08957356dfda4d475f8255291d7ba65266" gracePeriod=30
Oct 29 22:09:48.061416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:48.056848    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"]
Oct 29 22:09:48.061416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:48.057074    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd" podUID="76bc7ece-7212-40cb-954d-4304de2eaf2c" containerName="payload-pre-processing" containerID="cri-o://2b713f22e1020f3edbe13267229b9c51c7d186685624f793629ed40c162ad97c" gracePeriod=30
Oct 29 22:09:49.080211 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.073446    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"
Oct 29 22:09:49.083902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.083352    7082 generic.go:356] "Generic (PLEG): container finished" podID="08aebda0-8d4c-433a-b844-2845c9a35971" containerID="30728a04fe0467c1efa51df7b1a1dd08957356dfda4d475f8255291d7ba65266" exitCode=0
Oct 29 22:09:49.083902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.083461    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"
Oct 29 22:09:49.083902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.083706    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs" event={"ID":"08aebda0-8d4c-433a-b844-2845c9a35971","Type":"ContainerDied","Data":"30728a04fe0467c1efa51df7b1a1dd08957356dfda4d475f8255291d7ba65266"}
Oct 29 22:09:49.083902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.083760    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs" event={"ID":"08aebda0-8d4c-433a-b844-2845c9a35971","Type":"ContainerDied","Data":"64973ce93596ebc88b72f200ebb4a10221111bbeec7f8dc6b7ee186fc5eb4e96"}
Oct 29 22:09:49.083902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.083792    7082 scope.go:122] "RemoveContainer" containerID="30728a04fe0467c1efa51df7b1a1dd08957356dfda4d475f8255291d7ba65266"
Oct 29 22:09:49.084769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.084746    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"
Oct 29 22:09:49.090194 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.089170    7082 generic.go:356] "Generic (PLEG): container finished" podID="76bc7ece-7212-40cb-954d-4304de2eaf2c" containerID="2b713f22e1020f3edbe13267229b9c51c7d186685624f793629ed40c162ad97c" exitCode=0
Oct 29 22:09:49.090194 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.089460    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd" event={"ID":"76bc7ece-7212-40cb-954d-4304de2eaf2c","Type":"ContainerDied","Data":"2b713f22e1020f3edbe13267229b9c51c7d186685624f793629ed40c162ad97c"}
Oct 29 22:09:49.090194 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.089509    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd" event={"ID":"76bc7ece-7212-40cb-954d-4304de2eaf2c","Type":"ContainerDied","Data":"cc15ae79d60980d6696d102792e3d27249993fba9c16cd1b07f1423363d6e6fb"}
Oct 29 22:09:49.105700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.105611    7082 scope.go:122] "RemoveContainer" containerID="30728a04fe0467c1efa51df7b1a1dd08957356dfda4d475f8255291d7ba65266"
Oct 29 22:09:49.106071 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:09:49.106043    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30728a04fe0467c1efa51df7b1a1dd08957356dfda4d475f8255291d7ba65266\": container with ID starting with 30728a04fe0467c1efa51df7b1a1dd08957356dfda4d475f8255291d7ba65266 not found: ID does not exist" containerID="30728a04fe0467c1efa51df7b1a1dd08957356dfda4d475f8255291d7ba65266"
Oct 29 22:09:49.106181 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.106084    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30728a04fe0467c1efa51df7b1a1dd08957356dfda4d475f8255291d7ba65266"} err="failed to get container status \"30728a04fe0467c1efa51df7b1a1dd08957356dfda4d475f8255291d7ba65266\": rpc error: code = NotFound desc = could not find container \"30728a04fe0467c1efa51df7b1a1dd08957356dfda4d475f8255291d7ba65266\": container with ID starting with 30728a04fe0467c1efa51df7b1a1dd08957356dfda4d475f8255291d7ba65266 not found: ID does not exist"
Oct 29 22:09:49.106181 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.106106    7082 scope.go:122] "RemoveContainer" containerID="2b713f22e1020f3edbe13267229b9c51c7d186685624f793629ed40c162ad97c"
Oct 29 22:09:49.132185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.132158    7082 scope.go:122] "RemoveContainer" containerID="2b713f22e1020f3edbe13267229b9c51c7d186685624f793629ed40c162ad97c"
Oct 29 22:09:49.134090 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:09:49.134058    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b713f22e1020f3edbe13267229b9c51c7d186685624f793629ed40c162ad97c\": container with ID starting with 2b713f22e1020f3edbe13267229b9c51c7d186685624f793629ed40c162ad97c not found: ID does not exist" containerID="2b713f22e1020f3edbe13267229b9c51c7d186685624f793629ed40c162ad97c"
Oct 29 22:09:49.134212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.134099    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b713f22e1020f3edbe13267229b9c51c7d186685624f793629ed40c162ad97c"} err="failed to get container status \"2b713f22e1020f3edbe13267229b9c51c7d186685624f793629ed40c162ad97c\": rpc error: code = NotFound desc = could not find container \"2b713f22e1020f3edbe13267229b9c51c7d186685624f793629ed40c162ad97c\": container with ID starting with 2b713f22e1020f3edbe13267229b9c51c7d186685624f793629ed40c162ad97c not found: ID does not exist"
Oct 29 22:09:49.234351 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.233933    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08aebda0-8d4c-433a-b844-2845c9a35971-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08aebda0-8d4c-433a-b844-2845c9a35971-plugins-config-volume\") pod \"08aebda0-8d4c-433a-b844-2845c9a35971\" (UID: \"08aebda0-8d4c-433a-b844-2845c9a35971\") "
Oct 29 22:09:49.234351 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.234079    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08aebda0-8d4c-433a-b844-2845c9a35971-kube-api-access-qv46t\" (UniqueName: \"kubernetes.io/projected/08aebda0-8d4c-433a-b844-2845c9a35971-kube-api-access-qv46t\") pod \"08aebda0-8d4c-433a-b844-2845c9a35971\" (UID: \"08aebda0-8d4c-433a-b844-2845c9a35971\") "
Oct 29 22:09:49.234351 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.234189    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bc7ece-7212-40cb-954d-4304de2eaf2c-kube-api-access-gr8ks\" (UniqueName: \"kubernetes.io/projected/76bc7ece-7212-40cb-954d-4304de2eaf2c-kube-api-access-gr8ks\") pod \"76bc7ece-7212-40cb-954d-4304de2eaf2c\" (UID: \"76bc7ece-7212-40cb-954d-4304de2eaf2c\") "
Oct 29 22:09:49.235314 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.235021    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08aebda0-8d4c-433a-b844-2845c9a35971-plugins-config-volume" pod "08aebda0-8d4c-433a-b844-2845c9a35971" (UID: "08aebda0-8d4c-433a-b844-2845c9a35971"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:49.235314 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.235218    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bc7ece-7212-40cb-954d-4304de2eaf2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bc7ece-7212-40cb-954d-4304de2eaf2c-plugins-config-volume\") pod \"76bc7ece-7212-40cb-954d-4304de2eaf2c\" (UID: \"76bc7ece-7212-40cb-954d-4304de2eaf2c\") "
Oct 29 22:09:49.236752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.235944    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bc7ece-7212-40cb-954d-4304de2eaf2c-plugins-config-volume" pod "76bc7ece-7212-40cb-954d-4304de2eaf2c" (UID: "76bc7ece-7212-40cb-954d-4304de2eaf2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:49.236752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.236221    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08aebda0-8d4c-433a-b844-2845c9a35971-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:49.236752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.236246    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76bc7ece-7212-40cb-954d-4304de2eaf2c-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:49.237946 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.237919    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bc7ece-7212-40cb-954d-4304de2eaf2c-kube-api-access-gr8ks" pod "76bc7ece-7212-40cb-954d-4304de2eaf2c" (UID: "76bc7ece-7212-40cb-954d-4304de2eaf2c"). InnerVolumeSpecName "kube-api-access-gr8ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:49.239282 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.239251    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08aebda0-8d4c-433a-b844-2845c9a35971-kube-api-access-qv46t" pod "08aebda0-8d4c-433a-b844-2845c9a35971" (UID: "08aebda0-8d4c-433a-b844-2845c9a35971"). InnerVolumeSpecName "kube-api-access-qv46t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:49.253376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.253348    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"]
Oct 29 22:09:49.323445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.276267    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dd38c2-89zjg"]
Oct 29 22:09:49.362412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.362182    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv46t\" (UniqueName: \"kubernetes.io/projected/08aebda0-8d4c-433a-b844-2845c9a35971-kube-api-access-qv46t\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:49.362412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.362222    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr8ks\" (UniqueName: \"kubernetes.io/projected/76bc7ece-7212-40cb-954d-4304de2eaf2c-kube-api-access-gr8ks\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:09:49.476860 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.476787    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"]
Oct 29 22:09:49.486699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.486617    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dd38c2-7bb7c8d8b6-rcrxs"]
Oct 29 22:09:49.667669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.667557    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08aebda0-8d4c-433a-b844-2845c9a35971" path="/var/lib/kubelet/pods/08aebda0-8d4c-433a-b844-2845c9a35971/volumes"
Oct 29 22:09:49.668420 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.668392    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8276481-b6a7-40d9-b2f1-99c1d8de9bb1" path="/var/lib/kubelet/pods/a8276481-b6a7-40d9-b2f1-99c1d8de9bb1/volumes"
Oct 29 22:09:49.791494 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:49.791115    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd" podUID="76bc7ece-7212-40cb-954d-4304de2eaf2c" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.160:9004: i/o timeout"
Oct 29 22:09:50.131315 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.131283    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"
Oct 29 22:09:50.192117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.192074    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"]
Oct 29 22:09:50.204455 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.204421    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dd38c2-56f5648bb7-hmlwd"]
Oct 29 22:09:50.649685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.649653    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"]
Oct 29 22:09:50.650473 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.650458    7082 cpu_manager.go:414] "containerMap: removing container" podUID="76bc7ece-7212-40cb-954d-4304de2eaf2c" containerName="payload-pre-processing"
Oct 29 22:09:50.650535 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.650475    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bc7ece-7212-40cb-954d-4304de2eaf2c" containerName="payload-pre-processing"
Oct 29 22:09:50.650535 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.650505    7082 cpu_manager.go:414] "containerMap: removing container" podUID="08aebda0-8d4c-433a-b844-2845c9a35971" containerName="payload-processing"
Oct 29 22:09:50.650535 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.650510    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08aebda0-8d4c-433a-b844-2845c9a35971" containerName="payload-processing"
Oct 29 22:09:50.650703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.650611    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bc7ece-7212-40cb-954d-4304de2eaf2c" containerName="payload-pre-processing"
Oct 29 22:09:50.650703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.650636    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="08aebda0-8d4c-433a-b844-2845c9a35971" containerName="payload-processing"
Oct 29 22:09:50.735461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.732816    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"]
Oct 29 22:09:50.735461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.732958    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"
Oct 29 22:09:50.895008 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.871983    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-27cda7\""
Oct 29 22:09:50.895008 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.872152    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-27cda7-dockercfg-nzzsz\""
Oct 29 22:09:50.937268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.905296    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"]
Oct 29 22:09:50.956682 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.950419    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s\" (UID: \"7decf11f-4aa4-475a-8eea-e6bfb17fd48f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"
Oct 29 22:09:50.956682 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:50.950702    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q259q\" (UniqueName: \"kubernetes.io/projected/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-kube-api-access-q259q\") pod \"payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s\" (UID: \"7decf11f-4aa4-475a-8eea-e6bfb17fd48f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"
Oct 29 22:09:51.089666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.079839    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s\" (UID: \"7decf11f-4aa4-475a-8eea-e6bfb17fd48f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"
Oct 29 22:09:51.089666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.085269    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s\" (UID: \"7decf11f-4aa4-475a-8eea-e6bfb17fd48f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"
Oct 29 22:09:51.119237 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.092898    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q259q\" (UniqueName: \"kubernetes.io/projected/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-kube-api-access-q259q\") pod \"payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s\" (UID: \"7decf11f-4aa4-475a-8eea-e6bfb17fd48f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"
Oct 29 22:09:51.193761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.181915    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"]
Oct 29 22:09:51.193761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.182077    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"
Oct 29 22:09:51.195548 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.195379    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q259q\" (UniqueName: \"kubernetes.io/projected/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-kube-api-access-q259q\") pod \"payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s\" (UID: \"7decf11f-4aa4-475a-8eea-e6bfb17fd48f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"
Oct 29 22:09:51.280835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.280802    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mzd\" (UniqueName: \"kubernetes.io/projected/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-kube-api-access-m8mzd\") pod \"payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj\" (UID: \"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"
Oct 29 22:09:51.291012 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.290972    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj\" (UID: \"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"
Oct 29 22:09:51.392981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.392934    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj\" (UID: \"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"
Oct 29 22:09:51.393241 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.393224    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mzd\" (UniqueName: \"kubernetes.io/projected/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-kube-api-access-m8mzd\") pod \"payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj\" (UID: \"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"
Oct 29 22:09:51.393919 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.393886    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj\" (UID: \"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"
Oct 29 22:09:51.418825 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.417450    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mzd\" (UniqueName: \"kubernetes.io/projected/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-kube-api-access-m8mzd\") pod \"payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj\" (UID: \"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"
Oct 29 22:09:51.461299 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.461226    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"
Oct 29 22:09:51.666193 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.666155    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bc7ece-7212-40cb-954d-4304de2eaf2c" path="/var/lib/kubelet/pods/76bc7ece-7212-40cb-954d-4304de2eaf2c/volumes"
Oct 29 22:09:51.693251 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.693216    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"
Oct 29 22:09:51.967260 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:51.967227    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"]
Oct 29 22:09:51.978722 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:09:51.978690    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7decf11f_4aa4_475a_8eea_e6bfb17fd48f.slice/crio-222938c9d4fca66296d4bdb77b360deee9ccae399a4268663a439c9637a52faf WatchSource:0}: Error finding container 222938c9d4fca66296d4bdb77b360deee9ccae399a4268663a439c9637a52faf: Status 404 returned error can't find the container with id 222938c9d4fca66296d4bdb77b360deee9ccae399a4268663a439c9637a52faf
Oct 29 22:09:52.623461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:52.612800    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s" event={"ID":"7decf11f-4aa4-475a-8eea-e6bfb17fd48f","Type":"ContainerStarted","Data":"222938c9d4fca66296d4bdb77b360deee9ccae399a4268663a439c9637a52faf"}
Oct 29 22:09:52.623461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:52.619036    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"]
Oct 29 22:09:52.725792 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:52.725758    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:52.725932 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:52.725814    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv" podUID="1fa3614e-71e6-4d35-9046-8876a080c622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:52.734791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:52.734759    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:52.734934 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:52.734802    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:53.616811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:53.616771    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:53.617085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:53.616843    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn" podUID="d54282f6-acb0-4f9d-9f59-2888caddb88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:53.637518 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:53.637484    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"
Oct 29 22:09:53.638040 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:53.637543    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj" event={"ID":"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907","Type":"ContainerStarted","Data":"b6c6ab8861412cfadababf6e115cb1960204cdb361997834c75b63ac2f993a65"}
Oct 29 22:09:53.638040 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:53.637587    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"
Oct 29 22:09:53.638040 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:53.637641    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj" event={"ID":"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907","Type":"ContainerStarted","Data":"6e4862ac2389306693f2fc4ef13bf4a067343d15b72d5c025f387df55473dc41"}
Oct 29 22:09:53.638040 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:53.637666    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s" event={"ID":"7decf11f-4aa4-475a-8eea-e6bfb17fd48f","Type":"ContainerStarted","Data":"7a1b522104e3112c45ce6c539fbde2e591a9f93b88511b74d772323637847393"}
Oct 29 22:09:53.772550 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:53.772475    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj" podStartSLOduration=3.772452802 podStartE2EDuration="3.772452802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:53.696411753 +0000 UTC m=+1366.196454065" watchObservedRunningTime="2026-10-29 22:09:53.772452802 +0000 UTC m=+1366.272495102"
Oct 29 22:09:53.779080 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:53.779022    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s" podStartSLOduration=3.779004525 podStartE2EDuration="3.779004525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:53.764568075 +0000 UTC m=+1366.264610376" watchObservedRunningTime="2026-10-29 22:09:53.779004525 +0000 UTC m=+1366.279046825"
Oct 29 22:09:57.274738 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:57.274686    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"]
Oct 29 22:09:57.439978 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:57.439936    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv" podUID="1fa3614e-71e6-4d35-9046-8876a080c622" containerName="istio-proxy" containerID="cri-o://4579e74b67898eb26085361037e2e59454d627c36403139e47a7811b3f8f1c18" gracePeriod=30
Oct 29 22:09:57.694679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:57.575742    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"]
Oct 29 22:09:57.803654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:57.782761    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" podUID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" containerName="istio-proxy" containerID="cri-o://409b522d598d1392f782b1e8d7576d9f96260387cdca9991c67083b6e5389ea5" gracePeriod=30
Oct 29 22:09:58.345651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:58.343757    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"]
Oct 29 22:09:58.345651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:58.344484    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy" containerID="cri-o://f0f462309de1e237e69e18727b7d47a0c7131b2f039708f08a868f8210cf6253" gracePeriod=30
Oct 29 22:09:58.546167 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:58.546127    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:58.551733 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:58.551677    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:58.967954 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:58.967917    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"]
Oct 29 22:09:59.160752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.159686    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"]
Oct 29 22:09:59.160752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.159835    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"
Oct 29 22:09:59.175502 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.174101    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"
Oct 29 22:09:59.188368 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.188175    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:09:59.188368 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.188248    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:09:59.188368 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.188336    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:09:59.191287 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.191227    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"]
Oct 29 22:09:59.191642 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.191588    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:09:59.210710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.198030    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-301fcc-2zsn9"]
Oct 29 22:09:59.255064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.255030    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg\" (UID: \"87ccfa60-2e27-4624-9a19-54bd96de798c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"
Oct 29 22:09:59.263498 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.263465    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg\" (UID: \"87ccfa60-2e27-4624-9a19-54bd96de798c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"
Oct 29 22:09:59.265064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.265030    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxw2r\" (UniqueName: \"kubernetes.io/projected/87ccfa60-2e27-4624-9a19-54bd96de798c-kube-api-access-cxw2r\") pod \"maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg\" (UID: \"87ccfa60-2e27-4624-9a19-54bd96de798c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"
Oct 29 22:09:59.337534 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:09:59.335669    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:09:59.366863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.366680    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg\" (UID: \"87ccfa60-2e27-4624-9a19-54bd96de798c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"
Oct 29 22:09:59.366863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.366819    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxw2r\" (UniqueName: \"kubernetes.io/projected/87ccfa60-2e27-4624-9a19-54bd96de798c-kube-api-access-cxw2r\") pod \"maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg\" (UID: \"87ccfa60-2e27-4624-9a19-54bd96de798c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"
Oct 29 22:09:59.367361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.367108    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg\" (UID: \"87ccfa60-2e27-4624-9a19-54bd96de798c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"
Oct 29 22:09:59.368506 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.368453    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg\" (UID: \"87ccfa60-2e27-4624-9a19-54bd96de798c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"
Oct 29 22:09:59.458390 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.458351    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxw2r\" (UniqueName: \"kubernetes.io/projected/87ccfa60-2e27-4624-9a19-54bd96de798c-kube-api-access-cxw2r\") pod \"maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg\" (UID: \"87ccfa60-2e27-4624-9a19-54bd96de798c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"
Oct 29 22:09:59.459451 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.459417    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg\" (UID: \"87ccfa60-2e27-4624-9a19-54bd96de798c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"
Oct 29 22:09:59.648983 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.647830    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="517d67a8-859e-473c-a71f-b2e83ca9077a" path="/var/lib/kubelet/pods/517d67a8-859e-473c-a71f-b2e83ca9077a/volumes"
Oct 29 22:09:59.659678 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:09:59.659461    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"
Oct 29 22:10:01.930012 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:01.929982    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"]
Oct 29 22:10:02.246392 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:02.246244    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg" event={"ID":"87ccfa60-2e27-4624-9a19-54bd96de798c","Type":"ContainerStarted","Data":"e05c66dbdb0013c7ab661ac42fe93a04e2b0261f1b7afa65a2b477a145a8bb0c"}
Oct 29 22:10:03.136595 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.134685    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"]
Oct 29 22:10:03.162644 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.155002    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.174360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.174324    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"]
Oct 29 22:10:03.261870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.261830    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg" event={"ID":"87ccfa60-2e27-4624-9a19-54bd96de798c","Type":"ContainerStarted","Data":"a5a23e7a6bcb79cc85f3e4ae38794e94f6ca87e72c1d84fa3cbacd58e08568ee"}
Oct 29 22:10:03.413799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.362789    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.413799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.362886    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.413799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.363015    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.413799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.363051    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.413799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.363120    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.413799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.363149    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.413799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.363208    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9j4\" (UniqueName: \"kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-kube-api-access-vf9j4\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.413799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.363328    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.413799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.363433    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.432080 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.432001    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg" podStartSLOduration=6.431980411 podStartE2EDuration="6.431980411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:03.345228314 +0000 UTC m=+1375.845270613" watchObservedRunningTime="2026-10-29 22:10:03.431980411 +0000 UTC m=+1375.932022711"
Oct 29 22:10:03.488235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.482483    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.488235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.482576    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.488235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.482677    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.488235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.482710    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.488235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.482797    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9j4\" (UniqueName: \"kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-kube-api-access-vf9j4\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.488235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.482925    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.488235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.483033    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.488235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.483139    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.488235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.483239    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.488235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.484495    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.488235 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.485896    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.507659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.506252    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.507659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.506595    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.527653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.522953    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.572695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.569687    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.613187 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.613153    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.642443 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.642065    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9j4\" (UniqueName: \"kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-kube-api-access-vf9j4\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.688653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.668578    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:03.854652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:03.848865    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:04.061584 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.058791    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"]
Oct 29 22:10:04.143702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.124868    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.356811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.356777    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.357132 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.357110    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.357441 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.357424    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.357677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.357662    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.357837 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.357822    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.358051 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.358036    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4r78\" (UniqueName: \"kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-kube-api-access-p4r78\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.358192 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.358178    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.358403 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.358388    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.358713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.358682    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de1a39a-2b26-4743-ab96-caa204152c4b-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.446748 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.446712    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"]
Oct 29 22:10:04.460440 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.460034    7082 generic.go:356] "Generic (PLEG): container finished" podID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" containerID="409b522d598d1392f782b1e8d7576d9f96260387cdca9991c67083b6e5389ea5" exitCode=0
Oct 29 22:10:04.460440 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.460373    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" event={"ID":"192f4565-6fc9-4fde-9f70-bc9da0b26a07","Type":"ContainerDied","Data":"409b522d598d1392f782b1e8d7576d9f96260387cdca9991c67083b6e5389ea5"}
Oct 29 22:10:04.460617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.460546    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de1a39a-2b26-4743-ab96-caa204152c4b-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.461164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.460703    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.461164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.460818    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.461164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.460946    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.461164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.461032    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.461164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.461078    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.461516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.461190    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4r78\" (UniqueName: \"kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-kube-api-access-p4r78\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.461516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.461254    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.461516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.461382    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.472230 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.470945    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.472230 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.471544    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de1a39a-2b26-4743-ab96-caa204152c4b-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.472688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.472546    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.472688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.472652    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.478084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.476204    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.480972 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.480937    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.504697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.484733    7082 generic.go:356] "Generic (PLEG): container finished" podID="87ccfa60-2e27-4624-9a19-54bd96de798c" containerID="a5a23e7a6bcb79cc85f3e4ae38794e94f6ca87e72c1d84fa3cbacd58e08568ee" exitCode=0
Oct 29 22:10:04.504697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.484842    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.504697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.484892    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg" event={"ID":"87ccfa60-2e27-4624-9a19-54bd96de798c","Type":"ContainerDied","Data":"a5a23e7a6bcb79cc85f3e4ae38794e94f6ca87e72c1d84fa3cbacd58e08568ee"}
Oct 29 22:10:04.544351 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.544027    7082 generic.go:356] "Generic (PLEG): container finished" podID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerID="f0f462309de1e237e69e18727b7d47a0c7131b2f039708f08a868f8210cf6253" exitCode=0
Oct 29 22:10:04.544351 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.544154    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" event={"ID":"5f7332e1-ce42-4071-bf1a-ddf706996684","Type":"ContainerDied","Data":"f0f462309de1e237e69e18727b7d47a0c7131b2f039708f08a868f8210cf6253"}
Oct 29 22:10:04.544351 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.544210    7082 scope.go:122] "RemoveContainer" containerID="a8703b676a08da768e9ed481426b8ca9ef2e64ac4db335f3e4d562eacf708558"
Oct 29 22:10:04.560763 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.560604    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4r78\" (UniqueName: \"kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-kube-api-access-p4r78\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.561439 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.561390    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:04.574383 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:10:04.574348    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fc8d32_632e_4ff3_90cb_f80fde70f134.slice/crio-07a5dfb8f04b7d6b66690be8533c57709697e02d38b9d6f52c0aea92fa66ddb1 WatchSource:0}: Error finding container 07a5dfb8f04b7d6b66690be8533c57709697e02d38b9d6f52c0aea92fa66ddb1: Status 404 returned error can't find the container with id 07a5dfb8f04b7d6b66690be8533c57709697e02d38b9d6f52c0aea92fa66ddb1
Oct 29 22:10:04.583662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.583123    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:04.583662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.583210    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:04.583662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.583251    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:04.612908 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.587505    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"]
Oct 29 22:10:04.645070 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.643907    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"
Oct 29 22:10:04.647105 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.646141    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"
Oct 29 22:10:04.763880 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:04.763852    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:05.432902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.419865    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:10:05.468217 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.468185    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-token\" (UniqueName: \"kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-token\") pod \"5f7332e1-ce42-4071-bf1a-ddf706996684\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") "
Oct 29 22:10:05.468689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.468612    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-kube-api-access-tt2n8\" (UniqueName: \"kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-kube-api-access-tt2n8\") pod \"5f7332e1-ce42-4071-bf1a-ddf706996684\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") "
Oct 29 22:10:05.469076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.469037    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-credential-socket\") pod \"5f7332e1-ce42-4071-bf1a-ddf706996684\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") "
Oct 29 22:10:05.469550 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.469317    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-credential-socket" pod "5f7332e1-ce42-4071-bf1a-ddf706996684" (UID: "5f7332e1-ce42-4071-bf1a-ddf706996684"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.469550 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.469393    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-certs\") pod \"5f7332e1-ce42-4071-bf1a-ddf706996684\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") "
Oct 29 22:10:05.469704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.469612    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-certs" pod "5f7332e1-ce42-4071-bf1a-ddf706996684" (UID: "5f7332e1-ce42-4071-bf1a-ddf706996684"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.469933 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.469912    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-data" pod "5f7332e1-ce42-4071-bf1a-ddf706996684" (UID: "5f7332e1-ce42-4071-bf1a-ddf706996684"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.470003 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.469951    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-data\") pod \"5f7332e1-ce42-4071-bf1a-ddf706996684\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") "
Oct 29 22:10:05.470068 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.470054    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-envoy\") pod \"5f7332e1-ce42-4071-bf1a-ddf706996684\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") "
Oct 29 22:10:05.470754 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.470730    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7332e1-ce42-4071-bf1a-ddf706996684-istiod-ca-cert" pod "5f7332e1-ce42-4071-bf1a-ddf706996684" (UID: "5f7332e1-ce42-4071-bf1a-ddf706996684"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:05.470832 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.470796    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f7332e1-ce42-4071-bf1a-ddf706996684-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7332e1-ce42-4071-bf1a-ddf706996684-istiod-ca-cert\") pod \"5f7332e1-ce42-4071-bf1a-ddf706996684\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") "
Oct 29 22:10:05.471393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.471375    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-podinfo\") pod \"5f7332e1-ce42-4071-bf1a-ddf706996684\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") "
Oct 29 22:10:05.471473 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.471459    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-socket\") pod \"5f7332e1-ce42-4071-bf1a-ddf706996684\" (UID: \"5f7332e1-ce42-4071-bf1a-ddf706996684\") "
Oct 29 22:10:05.471816 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.471796    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-socket" pod "5f7332e1-ce42-4071-bf1a-ddf706996684" (UID: "5f7332e1-ce42-4071-bf1a-ddf706996684"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.472800 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.472777    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.472895 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.472818    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.472895 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.472832    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.472895 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.472843    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7332e1-ce42-4071-bf1a-ddf706996684-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.472895 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.472854    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.560847 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.560667    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"
Oct 29 22:10:05.560847 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.560667    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt" event={"ID":"5f7332e1-ce42-4071-bf1a-ddf706996684","Type":"ContainerDied","Data":"2bbf80ae0ad0112fd0cd6e89aca45592d7257172af1122068fe30d687201047d"}
Oct 29 22:10:05.560847 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.560836    7082 scope.go:122] "RemoveContainer" containerID="f0f462309de1e237e69e18727b7d47a0c7131b2f039708f08a868f8210cf6253"
Oct 29 22:10:05.563503 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.563432    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn" event={"ID":"192f4565-6fc9-4fde-9f70-bc9da0b26a07","Type":"ContainerDied","Data":"ed94eb1b15bb06b4a2450ae114a7185ae318089a6bbf2029bdca00adc36a9088"}
Oct 29 22:10:05.563503 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.563474    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed94eb1b15bb06b4a2450ae114a7185ae318089a6bbf2029bdca00adc36a9088"
Oct 29 22:10:05.564780 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.564736    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-kube-api-access-tt2n8" pod "5f7332e1-ce42-4071-bf1a-ddf706996684" (UID: "5f7332e1-ce42-4071-bf1a-ddf706996684"). InnerVolumeSpecName "kube-api-access-tt2n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:05.565653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.565500    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-podinfo" pod "5f7332e1-ce42-4071-bf1a-ddf706996684" (UID: "5f7332e1-ce42-4071-bf1a-ddf706996684"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:05.566189 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.565772    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-token" pod "5f7332e1-ce42-4071-bf1a-ddf706996684" (UID: "5f7332e1-ce42-4071-bf1a-ddf706996684"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:05.566568 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.566471    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" event={"ID":"e6fc8d32-632e-4ff3-90cb-f80fde70f134","Type":"ContainerStarted","Data":"8ea8f72cc4dfa211d66730ae265552c355d8e2a4f2707b27b3f93e555afcd216"}
Oct 29 22:10:05.566568 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.566512    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" event={"ID":"e6fc8d32-632e-4ff3-90cb-f80fde70f134","Type":"ContainerStarted","Data":"07a5dfb8f04b7d6b66690be8533c57709697e02d38b9d6f52c0aea92fa66ddb1"}
Oct 29 22:10:05.566568 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.566542    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-envoy" pod "5f7332e1-ce42-4071-bf1a-ddf706996684" (UID: "5f7332e1-ce42-4071-bf1a-ddf706996684"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.575291 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.575212    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.575291 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.575240    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.575291 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.575256    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt2n8\" (UniqueName: \"kubernetes.io/projected/5f7332e1-ce42-4071-bf1a-ddf706996684-kube-api-access-tt2n8\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.575291 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.575271    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7332e1-ce42-4071-bf1a-ddf706996684-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.580756 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:10:05.580731    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de1a39a_2b26_4743_ab96_caa204152c4b.slice/crio-281602ca0fabe632484f9cc6d66211cf1e5e99cfe67899d4d3eff2bbcaa14694 WatchSource:0}: Error finding container 281602ca0fabe632484f9cc6d66211cf1e5e99cfe67899d4d3eff2bbcaa14694: Status 404 returned error can't find the container with id 281602ca0fabe632484f9cc6d66211cf1e5e99cfe67899d4d3eff2bbcaa14694
Oct 29 22:10:05.581292 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.581171    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"]
Oct 29 22:10:05.588802 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.585207    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:05.588802 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.585285    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:05.588802 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.585330    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:05.668416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.619738    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.676350    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-certs\") pod \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") "
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.676407    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-data\") pod \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") "
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.676469    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-podinfo\") pod \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") "
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.676517    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-socket\") pod \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") "
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.676579    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-kube-api-access-ljcbn\" (UniqueName: \"kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-kube-api-access-ljcbn\") pod \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") "
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.676809    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-envoy\") pod \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") "
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.676979    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istiod-ca-cert\") pod \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") "
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.677110    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-token\" (UniqueName: \"kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-token\") pod \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") "
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.677208    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-credential-socket\") pod \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\" (UID: \"192f4565-6fc9-4fde-9f70-bc9da0b26a07\") "
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.678556    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-socket" pod "192f4565-6fc9-4fde-9f70-bc9da0b26a07" (UID: "192f4565-6fc9-4fde-9f70-bc9da0b26a07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.678783    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-certs" pod "192f4565-6fc9-4fde-9f70-bc9da0b26a07" (UID: "192f4565-6fc9-4fde-9f70-bc9da0b26a07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.678977    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-data" pod "192f4565-6fc9-4fde-9f70-bc9da0b26a07" (UID: "192f4565-6fc9-4fde-9f70-bc9da0b26a07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.679556    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-credential-socket" pod "192f4565-6fc9-4fde-9f70-bc9da0b26a07" (UID: "192f4565-6fc9-4fde-9f70-bc9da0b26a07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.682673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.679906    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istiod-ca-cert" pod "192f4565-6fc9-4fde-9f70-bc9da0b26a07" (UID: "192f4565-6fc9-4fde-9f70-bc9da0b26a07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:05.701358 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.701297    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-podinfo" pod "192f4565-6fc9-4fde-9f70-bc9da0b26a07" (UID: "192f4565-6fc9-4fde-9f70-bc9da0b26a07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:05.702027 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.701997    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-token" pod "192f4565-6fc9-4fde-9f70-bc9da0b26a07" (UID: "192f4565-6fc9-4fde-9f70-bc9da0b26a07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:05.702140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.702050    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-kube-api-access-ljcbn" pod "192f4565-6fc9-4fde-9f70-bc9da0b26a07" (UID: "192f4565-6fc9-4fde-9f70-bc9da0b26a07"). InnerVolumeSpecName "kube-api-access-ljcbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:05.708493 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.708445    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-envoy" pod "192f4565-6fc9-4fde-9f70-bc9da0b26a07" (UID: "192f4565-6fc9-4fde-9f70-bc9da0b26a07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.780445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.780336    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.780445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.780365    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.780445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.780373    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.780445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.780386    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.780445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.780394    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.780445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.780402    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.780445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.780410    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljcbn\" (UniqueName: \"kubernetes.io/projected/192f4565-6fc9-4fde-9f70-bc9da0b26a07-kube-api-access-ljcbn\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.780445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.780419    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.780445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:05.780427    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192f4565-6fc9-4fde-9f70-bc9da0b26a07-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:06.023738 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.023128    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:10:06.145310 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.145280    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"
Oct 29 22:10:06.230274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.204504    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ccfa60-2e27-4624-9a19-54bd96de798c-kube-api-access-cxw2r\" (UniqueName: \"kubernetes.io/projected/87ccfa60-2e27-4624-9a19-54bd96de798c-kube-api-access-cxw2r\") pod \"87ccfa60-2e27-4624-9a19-54bd96de798c\" (UID: \"87ccfa60-2e27-4624-9a19-54bd96de798c\") "
Oct 29 22:10:06.230274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.204560    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-xks-ca\") pod \"87ccfa60-2e27-4624-9a19-54bd96de798c\" (UID: \"87ccfa60-2e27-4624-9a19-54bd96de798c\") "
Oct 29 22:10:06.230274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.204757    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-openshift-ca\") pod \"87ccfa60-2e27-4624-9a19-54bd96de798c\" (UID: \"87ccfa60-2e27-4624-9a19-54bd96de798c\") "
Oct 29 22:10:06.230274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.230226    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-openshift-ca" pod "87ccfa60-2e27-4624-9a19-54bd96de798c" (UID: "87ccfa60-2e27-4624-9a19-54bd96de798c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:06.284074 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.283978    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"]
Oct 29 22:10:06.299743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.295965    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-xks-ca" pod "87ccfa60-2e27-4624-9a19-54bd96de798c" (UID: "87ccfa60-2e27-4624-9a19-54bd96de798c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:06.330888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.310030    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:06.330888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.310062    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ccfa60-2e27-4624-9a19-54bd96de798c-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:06.330888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.330807    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ntgtt"]
Oct 29 22:10:06.538936 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.538903    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ccfa60-2e27-4624-9a19-54bd96de798c-kube-api-access-cxw2r" pod "87ccfa60-2e27-4624-9a19-54bd96de798c" (UID: "87ccfa60-2e27-4624-9a19-54bd96de798c"). InnerVolumeSpecName "kube-api-access-cxw2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:06.587338 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.587304    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg" event={"ID":"87ccfa60-2e27-4624-9a19-54bd96de798c","Type":"ContainerDied","Data":"e05c66dbdb0013c7ab661ac42fe93a04e2b0261f1b7afa65a2b477a145a8bb0c"}
Oct 29 22:10:06.587580 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.587561    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e05c66dbdb0013c7ab661ac42fe93a04e2b0261f1b7afa65a2b477a145a8bb0c"
Oct 29 22:10:06.587771 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.587757    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"
Oct 29 22:10:06.612656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.609904    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" event={"ID":"8de1a39a-2b26-4743-ab96-caa204152c4b","Type":"ContainerStarted","Data":"bb0d3970ad0284d3e5003a858cdde4a4b38ad19021a5f1fad440b4c751cde795"}
Oct 29 22:10:06.612656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.609962    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" event={"ID":"8de1a39a-2b26-4743-ab96-caa204152c4b","Type":"ContainerStarted","Data":"281602ca0fabe632484f9cc6d66211cf1e5e99cfe67899d4d3eff2bbcaa14694"}
Oct 29 22:10:06.612656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.610031    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"
Oct 29 22:10:06.637412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.625804    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxw2r\" (UniqueName: \"kubernetes.io/projected/87ccfa60-2e27-4624-9a19-54bd96de798c-kube-api-access-cxw2r\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:06.698076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.696750    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podStartSLOduration=4.696730959 podStartE2EDuration="4.696730959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:06.695470634 +0000 UTC m=+1379.195512934" watchObservedRunningTime="2026-10-29 22:10:06.696730959 +0000 UTC m=+1379.196773259"
Oct 29 22:10:06.764015 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.763981    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:06.811908 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.811773    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podStartSLOduration=3.811736964 podStartE2EDuration="3.811736964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:06.762537555 +0000 UTC m=+1379.262579854" watchObservedRunningTime="2026-10-29 22:10:06.811736964 +0000 UTC m=+1379.311779264"
Oct 29 22:10:06.819559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.819493    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"]
Oct 29 22:10:06.834868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.834784    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-tc6cn"]
Oct 29 22:10:06.864026 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.863488    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:06.864026 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.863547    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:06.864496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:06.864400    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:07.084991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:07.084904    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"]
Oct 29 22:10:07.085536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:07.085500    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy" containerID="cri-o://a0b166882f38cf78e3417e0d5a265fb267174641c1779a4abed798b168599f1c" gracePeriod=30
Oct 29 22:10:07.693654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:07.690141    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:07.693654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:07.690207    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv" podUID="1fa3614e-71e6-4d35-9046-8876a080c622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:07.718730 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:07.700585    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:07.718730 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:07.700686    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:07.750270 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:07.750233    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" path="/var/lib/kubelet/pods/192f4565-6fc9-4fde-9f70-bc9da0b26a07/volumes"
Oct 29 22:10:07.751425 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:07.751397    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" path="/var/lib/kubelet/pods/5f7332e1-ce42-4071-bf1a-ddf706996684/volumes"
Oct 29 22:10:07.779658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:07.776933    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:07.779658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:07.777023    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:07.850726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:07.850094    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:07.850726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:07.850177    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:08.253650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.246970    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"]
Oct 29 22:10:08.253650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.248406    7082 cpu_manager.go:414] "containerMap: removing container" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy"
Oct 29 22:10:08.253650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.248427    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy"
Oct 29 22:10:08.253650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.248458    7082 cpu_manager.go:414] "containerMap: removing container" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy"
Oct 29 22:10:08.253650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.248467    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy"
Oct 29 22:10:08.253650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.248527    7082 cpu_manager.go:414] "containerMap: removing container" podUID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" containerName="istio-proxy"
Oct 29 22:10:08.253650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.248534    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" containerName="istio-proxy"
Oct 29 22:10:08.253650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.248550    7082 cpu_manager.go:414] "containerMap: removing container" podUID="87ccfa60-2e27-4624-9a19-54bd96de798c" containerName="revoke-keys"
Oct 29 22:10:08.253650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.248558    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ccfa60-2e27-4624-9a19-54bd96de798c" containerName="revoke-keys"
Oct 29 22:10:08.253650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.248734    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy"
Oct 29 22:10:08.253650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.248756    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="192f4565-6fc9-4fde-9f70-bc9da0b26a07" containerName="istio-proxy"
Oct 29 22:10:08.253650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.248770    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ccfa60-2e27-4624-9a19-54bd96de798c" containerName="revoke-keys"
Oct 29 22:10:08.412660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.409607    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"]
Oct 29 22:10:08.412660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.409771    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"
Oct 29 22:10:08.435262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.427910    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:10:08.538663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.522466    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:10:08.591471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.590402    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:10:08.591471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.590500    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:10:08.625996 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.625366    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:08.625996 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.625436    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn" podUID="d54282f6-acb0-4f9d-9f59-2888caddb88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:08.715709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.711538    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7t9c\" (UniqueName: \"kubernetes.io/projected/35b469f3-9bf7-4da9-9c75-98b44db7c344-kube-api-access-t7t9c\") pod \"maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b\" (UID: \"35b469f3-9bf7-4da9-9c75-98b44db7c344\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"
Oct 29 22:10:08.715709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.711714    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b\" (UID: \"35b469f3-9bf7-4da9-9c75-98b44db7c344\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"
Oct 29 22:10:08.738196 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.727854    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b\" (UID: \"35b469f3-9bf7-4da9-9c75-98b44db7c344\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"
Oct 29 22:10:08.778258 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.772048    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:08.778258 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.772118    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:08.903872 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.903836    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:08.904153 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.904124    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:09.039526 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.983307    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7t9c\" (UniqueName: \"kubernetes.io/projected/35b469f3-9bf7-4da9-9c75-98b44db7c344-kube-api-access-t7t9c\") pod \"maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b\" (UID: \"35b469f3-9bf7-4da9-9c75-98b44db7c344\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"
Oct 29 22:10:09.039526 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.983481    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b\" (UID: \"35b469f3-9bf7-4da9-9c75-98b44db7c344\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"
Oct 29 22:10:09.039526 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.983587    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b\" (UID: \"35b469f3-9bf7-4da9-9c75-98b44db7c344\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"
Oct 29 22:10:09.039526 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:08.984646    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b\" (UID: \"35b469f3-9bf7-4da9-9c75-98b44db7c344\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"
Oct 29 22:10:09.175514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:09.132680    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b\" (UID: \"35b469f3-9bf7-4da9-9c75-98b44db7c344\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"
Oct 29 22:10:09.175514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:09.140362    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7t9c\" (UniqueName: \"kubernetes.io/projected/35b469f3-9bf7-4da9-9c75-98b44db7c344-kube-api-access-t7t9c\") pod \"maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b\" (UID: \"35b469f3-9bf7-4da9-9c75-98b44db7c344\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"
Oct 29 22:10:09.384332 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:09.362082    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"
Oct 29 22:10:09.815198 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:09.787679    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:09.815198 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:09.787749    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:09.864687 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:09.852325    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:09.864687 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:09.852388    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:10.373151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:10.373105    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:10:10.386241 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:10.385322    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:10:10.524361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:10.514184    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.598812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:10.598745    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.831707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:10.802849    7082 patch_prober.go:29] interesting pod/metrics-server-f7fc46b4-vvxlv container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:10250/livez\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:10.831707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:10.802935    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-f7fc46b4-vvxlv" podUID="baaab207-5534-4ef9-ac3a-9466ea5b900e" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.15:10250/livez\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.842943 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:10.842910    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:10.843186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:10.843162    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:10.956809 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:10.879723    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:10.956809 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:10.879780    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:10.956809 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:10.921704    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:10:11.133805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:11.130699    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.431359 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:11.384825    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:10:11.431359 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:11.386533    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:10:11.431359 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:11.415736    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.577650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:11.573882    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:11.577650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:11.574358    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:10:11.718708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:11.717069    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:11.746659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:11.722938    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.776654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:11.766252    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:11.776654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:11.766313    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:11.882190 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:11.882155    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:11.893706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:11.893665    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:11.948820 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:11.948770    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.119718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.042132    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:10:12.449769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.437537    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:10:12.449769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.437585    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:10:12.449769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.144444    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:10:12.449769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.209838    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.449769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.396332    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:10:12.449769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.439177    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:10:12.615654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.607714    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.641661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.641325    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:10:12.693708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.693215    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:12.693708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.693283    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.729260 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.718714    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:12.770962 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.724180    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.770962 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.769603    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:10:12.771742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.771585    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:12.771742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.771647    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:12.982730 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.898534    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:12.982730 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.898594    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:12.982730 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:12.960827    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.194532 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.193793    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:13.194532 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.193858    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.604704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.442128    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:10:13.604704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.468910    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.604704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.522578    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Oct 29 22:10:13.811185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.656253    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:13.811185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.681225    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:13.811185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.681280    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.811185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.697137    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.811185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.771816    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.811185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.781346    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.844604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.826685    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:13.844604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.826750    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:13.940063 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.921187    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:13.940063 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.921249    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:13.972780 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:13.972721    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.498650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:14.472727    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.908021 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:14.892118    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:14.908021 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:14.892205    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:14.927673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:14.909605    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.986835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:14.961024    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:15.042443 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:15.025024    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:15.063666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:15.062824    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:15.086200 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:14.995657    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:10:15.086200 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:15.086162    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:15.086503 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:15.086226    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:15.107700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:15.094498    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:10:15.181077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:15.156596    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": context deadline exceeded"
Oct 29 22:10:15.570942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:15.570899    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:15.910844 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:15.871824    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:15.910844 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:15.871888    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:15.991763 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:15.982333    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:16.015157 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.015110    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:16.076660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.048324    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:16.210776 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.052415    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs" podUID="c32b079b-eba6-4e6f-8f79-f9c3cb86de99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:10:16.230742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.101406    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.270651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.139466    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.272115 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.141177    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:16.305656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.143708    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.306743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.143782    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:10:16.399381 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:10:16.365511    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.741s"
Oct 29 22:10:16.437834 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.418880    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": context deadline exceeded"
Oct 29 22:10:16.767656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.767463    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:16.767656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.767522    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:16.796824 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.789287    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b" event={"ID":"35b469f3-9bf7-4da9-9c75-98b44db7c344","Type":"ContainerStarted","Data":"31171deeb363b082732df594846ae1651674cb2bf579e5511c19d7b97360e6ad"}
Oct 29 22:10:16.796824 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.793079    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.068726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:16.977901    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"]
Oct 29 22:10:17.070216 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.070169    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:17.070455 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.070427    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:17.400245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.400145    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.420281 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.420234    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.420686 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.420661    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.421023 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.421005    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.421890 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.421338    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.446077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.446039    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.447405 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.447369    7082 generic.go:356] "Generic (PLEG): container finished" podID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerID="a0b166882f38cf78e3417e0d5a265fb267174641c1779a4abed798b168599f1c" exitCode=0
Oct 29 22:10:17.447729 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.447663    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" event={"ID":"9ef300af-2bfa-4891-9c03-e60646549fc3","Type":"ContainerDied","Data":"a0b166882f38cf78e3417e0d5a265fb267174641c1779a4abed798b168599f1c"}
Oct 29 22:10:17.547373 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.547142    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:17.547373 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.547232    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.938653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.935495    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:17.938653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.935576    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:17.963886 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.956505    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.963886 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:17.957803    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.015192 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.012832    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:18.015192 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.012893    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:18.317751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.317712    7082 scope.go:122] "RemoveContainer" containerID="3f3cf7f4c8f056359379673e4821b910c35dc16e92827754b5193f25c47f731a"
Oct 29 22:10:18.642651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.607792    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.642651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.619944    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.642651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.620340    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.642651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.620648    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.642651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.620904    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.642651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.622271    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.642651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.622531    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt" podUID="f069baec-59d9-4c9e-b5a2-c25261d01cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.828653 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:10:18.810109    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.164s"
Oct 29 22:10:18.859088 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.859064    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"]
Oct 29 22:10:19.051479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.027789    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:19.051479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.858892    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" podUID="4684c6fd-8ac7-4005-a20d-48396c0ab981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.051479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.033003    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.051479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.813425    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:19.051479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.041902    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.051479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.821659    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:19.051479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.041986    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:19.051479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.902806    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:19.051479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.042023    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:19.089498 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.870858    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" podUID="0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.089498 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:18.949840    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:10:19.259153 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.249890    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"]
Oct 29 22:10:19.259153 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.250125    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z" podUID="995d93a0-d033-41c4-a7a0-1ca49c98efaf" containerName="payload-processing" containerID="cri-o://d5c80373c4035af159219c87f339ef11f4025fdfaa1971c506692751629119f1" gracePeriod=30
Oct 29 22:10:19.430769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.426526    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-credential-socket\") pod \"9ef300af-2bfa-4891-9c03-e60646549fc3\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") "
Oct 29 22:10:19.430769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.426596    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-certs\") pod \"9ef300af-2bfa-4891-9c03-e60646549fc3\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") "
Oct 29 22:10:19.430769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.426654    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-socket\") pod \"9ef300af-2bfa-4891-9c03-e60646549fc3\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") "
Oct 29 22:10:19.430769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.426776    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-kube-api-access-9szb2\" (UniqueName: \"kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-kube-api-access-9szb2\") pod \"9ef300af-2bfa-4891-9c03-e60646549fc3\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") "
Oct 29 22:10:19.430769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.426828    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-token\" (UniqueName: \"kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-token\") pod \"9ef300af-2bfa-4891-9c03-e60646549fc3\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") "
Oct 29 22:10:19.430769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.426860    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef300af-2bfa-4891-9c03-e60646549fc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef300af-2bfa-4891-9c03-e60646549fc3-istiod-ca-cert\") pod \"9ef300af-2bfa-4891-9c03-e60646549fc3\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") "
Oct 29 22:10:19.430769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.426994    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-podinfo\") pod \"9ef300af-2bfa-4891-9c03-e60646549fc3\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") "
Oct 29 22:10:19.430769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.427041    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-envoy\") pod \"9ef300af-2bfa-4891-9c03-e60646549fc3\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") "
Oct 29 22:10:19.430769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.427249    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-data\") pod \"9ef300af-2bfa-4891-9c03-e60646549fc3\" (UID: \"9ef300af-2bfa-4891-9c03-e60646549fc3\") "
Oct 29 22:10:19.446516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.433534    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-data" pod "9ef300af-2bfa-4891-9c03-e60646549fc3" (UID: "9ef300af-2bfa-4891-9c03-e60646549fc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:19.468763 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.468716    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.477989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.477943    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-certs" pod "9ef300af-2bfa-4891-9c03-e60646549fc3" (UID: "9ef300af-2bfa-4891-9c03-e60646549fc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:19.478519 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.478491    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-socket" pod "9ef300af-2bfa-4891-9c03-e60646549fc3" (UID: "9ef300af-2bfa-4891-9c03-e60646549fc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:19.533101 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.506777    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef300af-2bfa-4891-9c03-e60646549fc3-istiod-ca-cert" pod "9ef300af-2bfa-4891-9c03-e60646549fc3" (UID: "9ef300af-2bfa-4891-9c03-e60646549fc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:19.533101 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.506886    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:19.533101 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.506930    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.621365 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.595919    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-credential-socket" pod "9ef300af-2bfa-4891-9c03-e60646549fc3" (UID: "9ef300af-2bfa-4891-9c03-e60646549fc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:19.673887 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.652955    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:10:19.708558 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.707363    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:19.708558 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.707428    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.708558 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.653133    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:19.829284 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.653112    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:10:19.829284 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.653165    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:19.829284 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.653093    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:10:19.829284 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.824795    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.843540 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.843486    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:19.874525 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.874484    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.874819 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.853474    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:19.874976 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.874951    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:20.039743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.969771    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef300af-2bfa-4891-9c03-e60646549fc3-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:20.039743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.969808    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:20.039743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.969822    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:20.039743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.969838    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:20.039743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.969852    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:20.039743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.996765    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:20.039743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:19.996825    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:20.041451 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.041406    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-envoy" pod "9ef300af-2bfa-4891-9c03-e60646549fc3" (UID: "9ef300af-2bfa-4891-9c03-e60646549fc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:20.075671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.061380    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.128683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.120165    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:20.275911 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.195174    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded"
Oct 29 22:10:20.276206 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.211499    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-kube-api-access-9szb2" pod "9ef300af-2bfa-4891-9c03-e60646549fc3" (UID: "9ef300af-2bfa-4891-9c03-e60646549fc3"). InnerVolumeSpecName "kube-api-access-9szb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:20.343960 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.343555    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-podinfo" pod "9ef300af-2bfa-4891-9c03-e60646549fc3" (UID: "9ef300af-2bfa-4891-9c03-e60646549fc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:20.439252 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.439219    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"
Oct 29 22:10:20.457176 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.457139    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9szb2\" (UniqueName: \"kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-kube-api-access-9szb2\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:20.457424 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.457406    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:20.468935 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.468898    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj" event={"ID":"9ef300af-2bfa-4891-9c03-e60646549fc3","Type":"ContainerDied","Data":"1fd44327cbb545c4d41ade09713443d5c4753e9a506da6aeaee54fae7098ed47"}
Oct 29 22:10:20.469129 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.468962    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"]
Oct 29 22:10:20.469129 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.469012    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:20.469129 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.469038    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"]
Oct 29 22:10:20.482659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.470293    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy"
Oct 29 22:10:20.482915 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.482890    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy"
Oct 29 22:10:20.483386 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.483361    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" containerName="istio-proxy"
Oct 29 22:10:20.483522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.483509    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy"
Oct 29 22:10:20.511678 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.484877    7082 scope.go:122] "RemoveContainer" containerID="a0b166882f38cf78e3417e0d5a265fb267174641c1779a4abed798b168599f1c"
Oct 29 22:10:20.511678 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.485133    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf" podUID="5b3d8a1a-2b77-42cc-8726-bcda92ec29c1" containerName="payload-pre-processing" containerID="cri-o://08866d852f549d871dd2cc2e7570be2723ac6e4751e0cd846165a63ec6be6243" gracePeriod=30
Oct 29 22:10:20.533819 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.533725    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:20.678206 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.654414    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:20.678206 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.654488    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:20.678206 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.654611    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:20.680080 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.680044    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:20.680397 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.680370    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:20.683833 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.683803    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:20.684076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.684059    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:20.684777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.684255    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:20.684777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.684291    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5xpd\" (UniqueName: \"kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-kube-api-access-f5xpd\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:20.696575 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.685233    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.759907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.716098    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.759907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.757867    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.784881 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.779437    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:20.784881 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.779509    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:20.794658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.786403    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.104670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.788836    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.121560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.805975    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.121560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.806483    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"]
Oct 29 22:10:21.121560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.110857    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy"
Oct 29 22:10:21.121560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.110879    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy"
Oct 29 22:10:21.121560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.111133    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" containerName="istio-proxy"
Oct 29 22:10:21.123456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.827937    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.123456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.797088    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-token" pod "9ef300af-2bfa-4891-9c03-e60646549fc3" (UID: "9ef300af-2bfa-4891-9c03-e60646549fc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:21.123456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.875096    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:21.123456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.122958    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:21.123456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:20.874972    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.124550 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.065188    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.156211 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.156175    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.163912 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.163877    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.172868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.172834    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.173134 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.173113    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5xpd\" (UniqueName: \"kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-kube-api-access-f5xpd\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.173342 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.173324    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.173526 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.173510    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.189713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.176598    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.189713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.177151    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.189713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.177609    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.189713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.178617    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.189713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.188641    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"]
Oct 29 22:10:21.189713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.189219    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"]
Oct 29 22:10:21.189713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.188744    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.223653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.191161    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.238969 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.238309    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.238969 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.238567    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef300af-2bfa-4891-9c03-e60646549fc3-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:21.238969 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.238923    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.366039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.357358    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrt49\" (UniqueName: \"kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-kube-api-access-wrt49\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.366039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.357610    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.366039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.357805    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.366039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.357872    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.366039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.358003    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.366039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.358114    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.366039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.358148    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.366039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.358193    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4301312f-6325-4e8c-9df7-8e4153fbcea9-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.366039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.358247    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.407355 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.407228    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.418142 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.410923    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.426655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.420769    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5xpd\" (UniqueName: \"kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-kube-api-access-f5xpd\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:21.552651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.478532    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.552651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.478593    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.552651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.491426    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4301312f-6325-4e8c-9df7-8e4153fbcea9-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.552651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.491547    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.552651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.491653    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrt49\" (UniqueName: \"kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-kube-api-access-wrt49\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.552651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.491889    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.552651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.492085    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.552651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.492146    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.552651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.492275    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.630792 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.571311    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:21.777683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:21.724787    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:22.110474 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:22.110406    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:22.287518 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:22.166847    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:10:22.287518 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:22.170292    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:22.287518 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:22.170644    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:22.287518 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:22.285037    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:22.319360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:22.319262    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf" podUID="5b3d8a1a-2b77-42cc-8726-bcda92ec29c1" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.177:9004: connect: connection refused"
Oct 29 22:10:23.125656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.120401    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.125656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:22.407572    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:23.125656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:22.428386    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:23.125656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:22.503663    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:23.125656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.123004    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:23.125656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:22.503738    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:23.125656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.123069    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:23.125656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:22.951519    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"]
Oct 29 22:10:23.181708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.169354    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4301312f-6325-4e8c-9df7-8e4153fbcea9-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:23.229655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:22.376682    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Oct 29 22:10:23.229655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.211195    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.229655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.212076    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:23.229655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.083655    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:23.229655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.221090    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.229655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.221173    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.295652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.240140    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb" podUID="7b205272-e4c0-432c-860d-162d9e80f950" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 29 22:10:23.295652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.241403    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:23.295652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.241514    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:23.333665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.333600    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.347171 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.346161    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:23.645016 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.582297    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.645016 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.582413    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.645016 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.588256    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:23.645016 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.616754    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.962859 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.919998    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrt49\" (UniqueName: \"kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-kube-api-access-wrt49\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:24.333515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:23.975782    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.333515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.095733    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:24.333515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.095815    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:24.333515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.133728    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:24.333515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.174007    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:24.333515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.174070    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:24.333515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.180958    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:24.333515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.181030    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:24.333515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.236766    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Oct 29 22:10:24.333515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.277487    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.333515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.277644    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.347074 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.347026    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:24.364782 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.364745    7082 generic.go:356] "Generic (PLEG): container finished" podID="35b469f3-9bf7-4da9-9c75-98b44db7c344" containerID="17b43d19110d84be9fb43d8dd1a0a4ec04e47a2194bdc7e37b0857d88fb2d45b" exitCode=0
Oct 29 22:10:24.409893 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.379148    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.409893 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.401340    7082 generic.go:356] "Generic (PLEG): container finished" podID="995d93a0-d033-41c4-a7a0-1ca49c98efaf" containerID="d5c80373c4035af159219c87f339ef11f4025fdfaa1971c506692751629119f1" exitCode=0
Oct 29 22:10:24.449671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.417083    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.449671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.417222    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.449671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.417286    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbzvg\" (UniqueName: \"kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-kube-api-access-jbzvg\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.449671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.417440    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.449671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.417548    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.449671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.417693    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.449671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.417805    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.449671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.417906    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.449671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.425436    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"
Oct 29 22:10:24.523733 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.472065    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.625909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.484428    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.625909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.484899    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:24.625909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.485402    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.627491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.492825    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:24.627491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.626254    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn" podUID="d54282f6-acb0-4f9d-9f59-2888caddb88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:24.627491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.524135    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:24.628394 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.560116    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:24.628981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.628955    7082 generic.go:356] "Generic (PLEG): container finished" podID="5b3d8a1a-2b77-42cc-8726-bcda92ec29c1" containerID="08866d852f549d871dd2cc2e7570be2723ac6e4751e0cd846165a63ec6be6243" exitCode=0
Oct 29 22:10:24.629820 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.629795    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs" podUID="c32b079b-eba6-4e6f-8f79-f9c3cb86de99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:24.635293 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.635267    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b" event={"ID":"35b469f3-9bf7-4da9-9c75-98b44db7c344","Type":"ContainerDied","Data":"17b43d19110d84be9fb43d8dd1a0a4ec04e47a2194bdc7e37b0857d88fb2d45b"}
Oct 29 22:10:24.638080 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.638053    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"]
Oct 29 22:10:24.640963 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.640938    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z" event={"ID":"995d93a0-d033-41c4-a7a0-1ca49c98efaf","Type":"ContainerDied","Data":"d5c80373c4035af159219c87f339ef11f4025fdfaa1971c506692751629119f1"}
Oct 29 22:10:24.641145 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.641128    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"]
Oct 29 22:10:24.641278 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.641262    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf" event={"ID":"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1","Type":"ContainerDied","Data":"08866d852f549d871dd2cc2e7570be2723ac6e4751e0cd846165a63ec6be6243"}
Oct 29 22:10:24.646275 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.646249    7082 scope.go:122] "RemoveContainer" containerID="d5c80373c4035af159219c87f339ef11f4025fdfaa1971c506692751629119f1"
Oct 29 22:10:24.687956 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.687921    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.704742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.704699    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z" podUID="995d93a0-d033-41c4-a7a0-1ca49c98efaf" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.178:9004: i/o timeout"
Oct 29 22:10:24.708820 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.708781    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.775100 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.775052    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/995d93a0-d033-41c4-a7a0-1ca49c98efaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995d93a0-d033-41c4-a7a0-1ca49c98efaf-plugins-config-volume\") pod \"995d93a0-d033-41c4-a7a0-1ca49c98efaf\" (UID: \"995d93a0-d033-41c4-a7a0-1ca49c98efaf\") "
Oct 29 22:10:24.805777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.805739    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995d93a0-d033-41c4-a7a0-1ca49c98efaf-kube-api-access-nz5c8\" (UniqueName: \"kubernetes.io/projected/995d93a0-d033-41c4-a7a0-1ca49c98efaf-kube-api-access-nz5c8\") pod \"995d93a0-d033-41c4-a7a0-1ca49c98efaf\" (UID: \"995d93a0-d033-41c4-a7a0-1ca49c98efaf\") "
Oct 29 22:10:24.998309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.807062    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.998309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.851995    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbzvg\" (UniqueName: \"kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-kube-api-access-jbzvg\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.998309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.852179    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.998309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.852297    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.998309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.852422    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.998309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.852534    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.998309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.852653    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:24.998309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.848541    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" podUID="4684c6fd-8ac7-4005-a20d-48396c0ab981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.998309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.868892    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995d93a0-d033-41c4-a7a0-1ca49c98efaf-plugins-config-volume" pod "995d93a0-d033-41c4-a7a0-1ca49c98efaf" (UID: "995d93a0-d033-41c4-a7a0-1ca49c98efaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:25.323651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:24.721302    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.343694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.337656    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995d93a0-d033-41c4-a7a0-1ca49c98efaf-kube-api-access-nz5c8" pod "995d93a0-d033-41c4-a7a0-1ca49c98efaf" (UID: "995d93a0-d033-41c4-a7a0-1ca49c98efaf"). InnerVolumeSpecName "kube-api-access-nz5c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:25.561867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.353012    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:25.561867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.522422    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv" podUID="1fa3614e-71e6-4d35-9046-8876a080c622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:25.561867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.352967    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:25.561867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.352869    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:25.561867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.352921    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" podUID="0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.561867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.435754    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:25.561867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.550112    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:25.561867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.474829    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:25.561867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.362565    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" podUID="0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:25.561867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.414601    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:25.628303 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.451307    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:25.628303 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.453113    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/995d93a0-d033-41c4-a7a0-1ca49c98efaf-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:25.628303 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.435799    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:25.628303 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.627840    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:25.628303 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.511156    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": context deadline exceeded"
Oct 29 22:10:25.628303 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.521077    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt" podUID="f069baec-59d9-4c9e-b5a2-c25261d01cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:25.687653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.659062    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.760728 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.737788    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.760728 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.739015    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:25.760728 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.746416    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:25.760728 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.746478    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.969704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.961462    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:25.969704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.961524    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:26.007889 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.988048    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:26.007889 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:25.988119    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:26.057654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:26.037221    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz5c8\" (UniqueName: \"kubernetes.io/projected/995d93a0-d033-41c4-a7a0-1ca49c98efaf-kube-api-access-nz5c8\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:26.302289 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:26.120585    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df42c7-btlqg"]
Oct 29 22:10:26.642471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:26.428197    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"
Oct 29 22:10:26.642471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:26.450689    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z" event={"ID":"995d93a0-d033-41c4-a7a0-1ca49c98efaf","Type":"ContainerDied","Data":"466cfca416ed8e7084401ad0f67c81f0aa3f4e1733262ca856fa85bcc5b2edcc"}
Oct 29 22:10:26.642471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:26.457479    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:26.642471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:26.525340    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:26.642471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:26.568842    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:26.642471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:26.617771    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:10:26.741457 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:26.712716    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:26.822315 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:26.809732    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:26.849734 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:26.848200    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:10:26.939861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:26.939802    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:27.535951 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:27.350294    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:27.564738 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:27.536807    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:27.564738 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:27.537112    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:27.826805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:27.824007    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:28.227126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:27.859803    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:10:28.227126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:27.860166    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:10:28.227126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:27.983410    7082 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-wc4l2 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded" start-of-body=
Oct 29 22:10:28.227126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:27.983469    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2" podUID="4c2dd590-c639-42fe-8a02-4945b6403b3b" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.14:9443/health\": context deadline exceeded"
Oct 29 22:10:28.227126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:27.983874    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded"
Oct 29 22:10:28.227126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:27.996719    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:28.227126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:27.996786    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:28.227126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.066725    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:28.227126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.086941    7082 patch_prober.go:29] interesting pod/thanos-querier-f96f5cd85-t7ndb container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.13:9091/-/healthy\": context deadline exceeded" start-of-body=
Oct 29 22:10:28.227126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.086992    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb" podUID="b96d72b2-bc45-4962-b529-f08155ad0215" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/healthy\": context deadline exceeded"
Oct 29 22:10:28.227126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.087060    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:10:28.227126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.172219    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:28.227126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.172269    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:28.607704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.568048    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:28.607704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.568104    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:28.688801 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.623695    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:28.688801 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.623775    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:28.688801 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.636263    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:28.835801 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.826175    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:28.835801 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.826237    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:28.883244 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.871008    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:28.900251 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.885378    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:28.984032 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.956963    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:10:28.984032 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:28.957364    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:29.017534 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.016760    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:10:29.031577 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:10:29.028747    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:10:29.065812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.050048    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbzvg\" (UniqueName: \"kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-kube-api-access-jbzvg\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:29.065812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.056542    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:29.065812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.056780    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:10:29.104849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.086140    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:29.104849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.087562    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:29.306499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.167381    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:29.306499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.167440    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:29.306499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.196100    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt" podUID="f069baec-59d9-4c9e-b5a2-c25261d01cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:29.306499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.230272    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:29.337788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.336486    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"
Oct 29 22:10:29.445697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.434912    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ccfa60-2e27-4624-9a19-54bd96de798c" path="/var/lib/kubelet/pods/87ccfa60-2e27-4624-9a19-54bd96de798c/volumes"
Oct 29 22:10:29.445697 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:10:29.435421    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.367s"
Oct 29 22:10:29.445697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.435453    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf" event={"ID":"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1","Type":"ContainerDied","Data":"ae7dc48ac9af29cf9a8c2177bbfc69b2541101371bbec89a4b51a28afbf23759"}
Oct 29 22:10:29.445697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.435595    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:10:29.445697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.435617    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"]
Oct 29 22:10:29.445697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.435681    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:10:29.445697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.435712    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:10:29.445697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.436512    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-plugins-config-volume\") pod \"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1\" (UID: \"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1\") "
Oct 29 22:10:29.445697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.436607    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-kube-api-access-k2k75\" (UniqueName: \"kubernetes.io/projected/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-kube-api-access-k2k75\") pod \"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1\" (UID: \"5b3d8a1a-2b77-42cc-8726-bcda92ec29c1\") "
Oct 29 22:10:29.455536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.454241    7082 scope.go:122] "RemoveContainer" containerID="08866d852f549d871dd2cc2e7570be2723ac6e4751e0cd846165a63ec6be6243"
Oct 29 22:10:29.560456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.460833    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-plugins-config-volume" pod "5b3d8a1a-2b77-42cc-8726-bcda92ec29c1" (UID: "5b3d8a1a-2b77-42cc-8726-bcda92ec29c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:29.560456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.470379    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:29.560456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.482141    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-kube-api-access-k2k75" pod "5b3d8a1a-2b77-42cc-8726-bcda92ec29c1" (UID: "5b3d8a1a-2b77-42cc-8726-bcda92ec29c1"). InnerVolumeSpecName "kube-api-access-k2k75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:29.560456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.483442    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:10:29.636923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.585403    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:29.636923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.585509    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:29.636923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.585563    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:29.932421 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.932327    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:30.202706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:29.932984    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:30.372921 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:10:30.109157    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995d93a0_d033_41c4_a7a0_1ca49c98efaf.slice/crio-466cfca416ed8e7084401ad0f67c81f0aa3f4e1733262ca856fa85bcc5b2edcc\": RecentStats: unable to find data in memory cache]"
Oct 29 22:10:30.375018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.110821    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:30.384068 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.361670    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2k75\" (UniqueName: \"kubernetes.io/projected/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1-kube-api-access-k2k75\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:30.384222 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.363500    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:10:30.400740 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.363537    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:30.480245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.478536    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:30.526193 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.513554    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 29 22:10:30.526193 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.513643    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 29 22:10:30.755270 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:10:30.751393    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.019s"
Oct 29 22:10:30.755270 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.751442    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-301fcc-openshift-default-767d445955-5mlgj"]
Oct 29 22:10:30.867912 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.833905    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:30.909213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.870329    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Oct 29 22:10:30.909213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.870384    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Oct 29 22:10:30.909213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.882522    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d" event={"ID":"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85","Type":"ContainerStarted","Data":"e53e5484c0dfa2219aa071539d877e6f20012f9ccb02ca5a8ddca8ee7151b48c"}
Oct 29 22:10:30.909213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.887421    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"
Oct 29 22:10:30.909213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.889492    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"
Oct 29 22:10:30.909213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.899562    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"]
Oct 29 22:10:30.909213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.903704    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b" event={"ID":"35b469f3-9bf7-4da9-9c75-98b44db7c344","Type":"ContainerDied","Data":"31171deeb363b082732df594846ae1651674cb2bf579e5511c19d7b97360e6ad"}
Oct 29 22:10:30.909213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.903768    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31171deeb363b082732df594846ae1651674cb2bf579e5511c19d7b97360e6ad"
Oct 29 22:10:30.909213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.904132    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"
Oct 29 22:10:30.910498 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.910468    7082 generic.go:356] "Generic (PLEG): container finished" podID="1fa3614e-71e6-4d35-9046-8876a080c622" containerID="4579e74b67898eb26085361037e2e59454d627c36403139e47a7811b3f8f1c18" exitCode=137
Oct 29 22:10:30.910723 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.910704    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv" event={"ID":"1fa3614e-71e6-4d35-9046-8876a080c622","Type":"ContainerDied","Data":"4579e74b67898eb26085361037e2e59454d627c36403139e47a7811b3f8f1c18"}
Oct 29 22:10:30.911525 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.911506    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df42c7-79d869c684-6w82z"]
Oct 29 22:10:30.917644 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.917599    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-xks-ca\") pod \"35b469f3-9bf7-4da9-9c75-98b44db7c344\" (UID: \"35b469f3-9bf7-4da9-9c75-98b44db7c344\") "
Oct 29 22:10:30.917753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.917702    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b469f3-9bf7-4da9-9c75-98b44db7c344-kube-api-access-t7t9c\" (UniqueName: \"kubernetes.io/projected/35b469f3-9bf7-4da9-9c75-98b44db7c344-kube-api-access-t7t9c\") pod \"35b469f3-9bf7-4da9-9c75-98b44db7c344\" (UID: \"35b469f3-9bf7-4da9-9c75-98b44db7c344\") "
Oct 29 22:10:30.917833 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.917809    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-openshift-ca\") pod \"35b469f3-9bf7-4da9-9c75-98b44db7c344\" (UID: \"35b469f3-9bf7-4da9-9c75-98b44db7c344\") "
Oct 29 22:10:30.935409 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:30.933491    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"]
Oct 29 22:10:31.049743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.049046    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:10:31.052587 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.052563    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:10:31.062052 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.061618    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy" containerID="cri-o://8ea8f72cc4dfa211d66730ae265552c355d8e2a4f2707b27b3f93e555afcd216" gracePeriod=30
Oct 29 22:10:31.062840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.062819    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-openshift-ca" pod "35b469f3-9bf7-4da9-9c75-98b44db7c344" (UID: "35b469f3-9bf7-4da9-9c75-98b44db7c344"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:31.063035 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.063019    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-xks-ca" pod "35b469f3-9bf7-4da9-9c75-98b44db7c344" (UID: "35b469f3-9bf7-4da9-9c75-98b44db7c344"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:31.081988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.079919    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"]
Oct 29 22:10:31.081988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.079981    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"]
Oct 29 22:10:31.081988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.080154    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn" podUID="d54282f6-acb0-4f9d-9f59-2888caddb88f" containerName="istio-proxy" containerID="cri-o://87ebad69e4423d3dc1a0ebf93e8a1309aa3fb614af9a44879c5657a26ea861bf" gracePeriod=30
Oct 29 22:10:31.081988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.080379    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b469f3-9bf7-4da9-9c75-98b44db7c344-kube-api-access-t7t9c" pod "35b469f3-9bf7-4da9-9c75-98b44db7c344" (UID: "35b469f3-9bf7-4da9-9c75-98b44db7c344"). InnerVolumeSpecName "kube-api-access-t7t9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:31.081988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.081346    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:31.085357 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.085312    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"]
Oct 29 22:10:31.143392 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.143364    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:31.146694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.146669    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7t9c\" (UniqueName: \"kubernetes.io/projected/35b469f3-9bf7-4da9-9c75-98b44db7c344-kube-api-access-t7t9c\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:31.146862 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.146848    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35b469f3-9bf7-4da9-9c75-98b44db7c344-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:31.153809 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.153757    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:31.154362 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.154039    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:31.154720 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.154530    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:31.156706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.156498    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"]
Oct 29 22:10:31.158262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.158222    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:31.158329 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.158289    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:31.158381 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.158353    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:31.164816 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.164794    7082 cpu_manager.go:414] "containerMap: removing container" podUID="35b469f3-9bf7-4da9-9c75-98b44db7c344" containerName="revoke-keys"
Oct 29 22:10:31.165985 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.165967    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b469f3-9bf7-4da9-9c75-98b44db7c344" containerName="revoke-keys"
Oct 29 22:10:31.166205 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.166193    7082 cpu_manager.go:414] "containerMap: removing container" podUID="5b3d8a1a-2b77-42cc-8726-bcda92ec29c1" containerName="payload-pre-processing"
Oct 29 22:10:31.166305 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.166295    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3d8a1a-2b77-42cc-8726-bcda92ec29c1" containerName="payload-pre-processing"
Oct 29 22:10:31.166435 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.166425    7082 cpu_manager.go:414] "containerMap: removing container" podUID="995d93a0-d033-41c4-a7a0-1ca49c98efaf" containerName="payload-processing"
Oct 29 22:10:31.166531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.166521    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995d93a0-d033-41c4-a7a0-1ca49c98efaf" containerName="payload-processing"
Oct 29 22:10:31.166905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.166891    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3d8a1a-2b77-42cc-8726-bcda92ec29c1" containerName="payload-pre-processing"
Oct 29 22:10:31.167035 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.167026    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="995d93a0-d033-41c4-a7a0-1ca49c98efaf" containerName="payload-processing"
Oct 29 22:10:31.167146 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.167137    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b469f3-9bf7-4da9-9c75-98b44db7c344" containerName="revoke-keys"
Oct 29 22:10:31.323420 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.323384    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:10:31.323601 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.323430    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"]
Oct 29 22:10:31.323601 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.323531    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:31.323601 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.323563    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"]
Oct 29 22:10:31.328755 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.325159    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:10:31.328755 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.325273    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"
Oct 29 22:10:31.338742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.338677    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"]
Oct 29 22:10:31.343754 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.343731    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:10:31.343910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.343886    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:10:31.344017 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.343947    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:10:31.344017 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.343992    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:10:31.344171 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.344110    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:10:31.477659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.473226    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-socket\") pod \"1fa3614e-71e6-4d35-9046-8876a080c622\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") "
Oct 29 22:10:31.477659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.473278    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fa3614e-71e6-4d35-9046-8876a080c622-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa3614e-71e6-4d35-9046-8876a080c622-istio-podinfo\") pod \"1fa3614e-71e6-4d35-9046-8876a080c622\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") "
Oct 29 22:10:31.477659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.473313    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-credential-socket\") pod \"1fa3614e-71e6-4d35-9046-8876a080c622\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") "
Oct 29 22:10:31.477659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.473399    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-data\") pod \"1fa3614e-71e6-4d35-9046-8876a080c622\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") "
Oct 29 22:10:31.477659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.473533    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-envoy\") pod \"1fa3614e-71e6-4d35-9046-8876a080c622\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") "
Oct 29 22:10:31.477659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.473692    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-istio-token\" (UniqueName: \"kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-istio-token\") pod \"1fa3614e-71e6-4d35-9046-8876a080c622\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") "
Oct 29 22:10:31.483681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.478279    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-certs\") pod \"1fa3614e-71e6-4d35-9046-8876a080c622\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") "
Oct 29 22:10:31.483681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.478332    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-kube-api-access-77wbw\" (UniqueName: \"kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-kube-api-access-77wbw\") pod \"1fa3614e-71e6-4d35-9046-8876a080c622\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") "
Oct 29 22:10:31.483681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.478511    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa3614e-71e6-4d35-9046-8876a080c622-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa3614e-71e6-4d35-9046-8876a080c622-istiod-ca-cert\") pod \"1fa3614e-71e6-4d35-9046-8876a080c622\" (UID: \"1fa3614e-71e6-4d35-9046-8876a080c622\") "
Oct 29 22:10:31.483681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.479033    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc\" (UID: \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"
Oct 29 22:10:31.484339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.484308    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4nbm\" (UniqueName: \"kubernetes.io/projected/b48e7dc6-2291-4349-84a1-40c6aa46a6df-kube-api-access-j4nbm\") pod \"maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc\" (UID: \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"
Oct 29 22:10:31.502651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.485013    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc\" (UID: \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"
Oct 29 22:10:31.504090 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.503326    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-socket" pod "1fa3614e-71e6-4d35-9046-8876a080c622" (UID: "1fa3614e-71e6-4d35-9046-8876a080c622"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:31.523264 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.523228    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-credential-socket" pod "1fa3614e-71e6-4d35-9046-8876a080c622" (UID: "1fa3614e-71e6-4d35-9046-8876a080c622"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:31.523747 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.523720    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-data" pod "1fa3614e-71e6-4d35-9046-8876a080c622" (UID: "1fa3614e-71e6-4d35-9046-8876a080c622"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:31.525265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.525240    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-certs" pod "1fa3614e-71e6-4d35-9046-8876a080c622" (UID: "1fa3614e-71e6-4d35-9046-8876a080c622"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:31.542971 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.537714    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa3614e-71e6-4d35-9046-8876a080c622-istiod-ca-cert" pod "1fa3614e-71e6-4d35-9046-8876a080c622" (UID: "1fa3614e-71e6-4d35-9046-8876a080c622"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:31.559994 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.559944    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-envoy" pod "1fa3614e-71e6-4d35-9046-8876a080c622" (UID: "1fa3614e-71e6-4d35-9046-8876a080c622"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:31.616531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.615224    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc\" (UID: \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"
Oct 29 22:10:31.616531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.615469    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4nbm\" (UniqueName: \"kubernetes.io/projected/b48e7dc6-2291-4349-84a1-40c6aa46a6df-kube-api-access-j4nbm\") pod \"maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc\" (UID: \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"
Oct 29 22:10:31.616531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.615690    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc\" (UID: \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"
Oct 29 22:10:31.616531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.615864    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:31.616531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.615891    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa3614e-71e6-4d35-9046-8876a080c622-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:31.616531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.615905    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:31.616531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.615918    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:31.616531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.615934    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:31.616531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.615958    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa3614e-71e6-4d35-9046-8876a080c622-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:31.620763 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.617973    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc\" (UID: \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"
Oct 29 22:10:31.799879 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.757200    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-kube-api-access-77wbw" pod "1fa3614e-71e6-4d35-9046-8876a080c622" (UID: "1fa3614e-71e6-4d35-9046-8876a080c622"). InnerVolumeSpecName "kube-api-access-77wbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:31.799879 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.758800    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fa3614e-71e6-4d35-9046-8876a080c622-istio-podinfo" pod "1fa3614e-71e6-4d35-9046-8876a080c622" (UID: "1fa3614e-71e6-4d35-9046-8876a080c622"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:31.799879 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.758913    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-istio-token" pod "1fa3614e-71e6-4d35-9046-8876a080c622" (UID: "1fa3614e-71e6-4d35-9046-8876a080c622"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:31.822174 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.822101    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa3614e-71e6-4d35-9046-8876a080c622-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:31.822174 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.822133    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:31.822174 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.822147    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77wbw\" (UniqueName: \"kubernetes.io/projected/1fa3614e-71e6-4d35-9046-8876a080c622-kube-api-access-77wbw\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:31.871494 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.871414    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4nbm\" (UniqueName: \"kubernetes.io/projected/b48e7dc6-2291-4349-84a1-40c6aa46a6df-kube-api-access-j4nbm\") pod \"maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc\" (UID: \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"
Oct 29 22:10:31.880282 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.880246    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc\" (UID: \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"
Oct 29 22:10:31.910983 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.910945    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995d93a0-d033-41c4-a7a0-1ca49c98efaf" path="/var/lib/kubelet/pods/995d93a0-d033-41c4-a7a0-1ca49c98efaf/volumes"
Oct 29 22:10:31.926456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.926379    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef300af-2bfa-4891-9c03-e60646549fc3" path="/var/lib/kubelet/pods/9ef300af-2bfa-4891-9c03-e60646549fc3/volumes"
Oct 29 22:10:31.936910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.936655    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"]
Oct 29 22:10:31.936910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.936717    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-7654dd46dc-8s8xf"]
Oct 29 22:10:31.968315 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.968279    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"
Oct 29 22:10:31.974839 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.973738    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv" event={"ID":"1fa3614e-71e6-4d35-9046-8876a080c622","Type":"ContainerDied","Data":"3a70b1e96dc346e47b0a2c4d0bb124e385a302f455818e3899e69ce8fbbaf941"}
Oct 29 22:10:31.974839 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.973823    7082 scope.go:122] "RemoveContainer" containerID="4579e74b67898eb26085361037e2e59454d627c36403139e47a7811b3f8f1c18"
Oct 29 22:10:31.974839 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:31.974019    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"
Oct 29 22:10:32.037924 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:32.037265    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" event={"ID":"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5","Type":"ContainerStarted","Data":"4dc81ba12cbd7b2048e7cafff6f783ce0443ce6b9773534b87a229854bb76843"}
Oct 29 22:10:32.060257 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:32.060219    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" event={"ID":"4301312f-6325-4e8c-9df7-8e4153fbcea9","Type":"ContainerStarted","Data":"0ed37f2ddb432486002840eb644b9733e2a6271c98c5f7e9d38fbca2f0f8ba3a"}
Oct 29 22:10:32.191036 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:32.190964    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d" podUID="76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" containerName="istio-proxy" containerID="cri-o://1c229e641dd1e9f76e91dbe9df7ad745e8d3e22243f4951bc86ca738618701a9" gracePeriod=30
Oct 29 22:10:32.192135 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:32.192107    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d" event={"ID":"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85","Type":"ContainerStarted","Data":"1c229e641dd1e9f76e91dbe9df7ad745e8d3e22243f4951bc86ca738618701a9"}
Oct 29 22:10:32.196262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:32.196237    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"]
Oct 29 22:10:32.205990 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:32.205923    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ls4mv"]
Oct 29 22:10:32.211998 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:32.210368    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" podStartSLOduration=14.210350985 podStartE2EDuration="14.210350985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:32.155641171 +0000 UTC m=+1404.655683462" watchObservedRunningTime="2026-10-29 22:10:32.210350985 +0000 UTC m=+1404.710393284"
Oct 29 22:10:32.456529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:32.456365    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:32.458358 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:32.456855    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 29 22:10:32.458358 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:32.456907    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 29 22:10:32.465001 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:32.463851    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d" podStartSLOduration=14.463833923 podStartE2EDuration="14.463833923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:32.260308229 +0000 UTC m=+1404.760350533" watchObservedRunningTime="2026-10-29 22:10:32.463833923 +0000 UTC m=+1404.963876224"
Oct 29 22:10:32.469844 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:32.469817    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"]
Oct 29 22:10:32.496216 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:10:32.496132    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48e7dc6_2291_4349_84a1_40c6aa46a6df.slice/crio-85c64e08a0cbfabe224fc2862206707f1d12ad846603b6d57ec6a6cc03e9d8ef WatchSource:0}: Error finding container 85c64e08a0cbfabe224fc2862206707f1d12ad846603b6d57ec6a6cc03e9d8ef: Status 404 returned error can't find the container with id 85c64e08a0cbfabe224fc2862206707f1d12ad846603b6d57ec6a6cc03e9d8ef
Oct 29 22:10:32.966236 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:32.966199    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:10:33.069463 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.068824    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-certs\") pod \"d54282f6-acb0-4f9d-9f59-2888caddb88f\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") "
Oct 29 22:10:33.069688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.069477    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-certs" pod "d54282f6-acb0-4f9d-9f59-2888caddb88f" (UID: "d54282f6-acb0-4f9d-9f59-2888caddb88f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.069782 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.069687    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d54282f6-acb0-4f9d-9f59-2888caddb88f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54282f6-acb0-4f9d-9f59-2888caddb88f-istiod-ca-cert\") pod \"d54282f6-acb0-4f9d-9f59-2888caddb88f\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") "
Oct 29 22:10:33.069782 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.069762    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-token\" (UniqueName: \"kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-token\") pod \"d54282f6-acb0-4f9d-9f59-2888caddb88f\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") "
Oct 29 22:10:33.069897 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.069834    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-credential-socket\") pod \"d54282f6-acb0-4f9d-9f59-2888caddb88f\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") "
Oct 29 22:10:33.069897 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.069884    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-envoy\") pod \"d54282f6-acb0-4f9d-9f59-2888caddb88f\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") "
Oct 29 22:10:33.070003 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.069959    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-socket\") pod \"d54282f6-acb0-4f9d-9f59-2888caddb88f\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") "
Oct 29 22:10:33.070063 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.070023    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-podinfo\") pod \"d54282f6-acb0-4f9d-9f59-2888caddb88f\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") "
Oct 29 22:10:33.070063 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.070057    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-data\") pod \"d54282f6-acb0-4f9d-9f59-2888caddb88f\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") "
Oct 29 22:10:33.070168 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.070091    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-kube-api-access-v2klk\" (UniqueName: \"kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-kube-api-access-v2klk\") pod \"d54282f6-acb0-4f9d-9f59-2888caddb88f\" (UID: \"d54282f6-acb0-4f9d-9f59-2888caddb88f\") "
Oct 29 22:10:33.072191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.071991    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-data" pod "d54282f6-acb0-4f9d-9f59-2888caddb88f" (UID: "d54282f6-acb0-4f9d-9f59-2888caddb88f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.072327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.072195    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-credential-socket" pod "d54282f6-acb0-4f9d-9f59-2888caddb88f" (UID: "d54282f6-acb0-4f9d-9f59-2888caddb88f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.072456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.072351    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-socket" pod "d54282f6-acb0-4f9d-9f59-2888caddb88f" (UID: "d54282f6-acb0-4f9d-9f59-2888caddb88f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.072869 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.072827    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54282f6-acb0-4f9d-9f59-2888caddb88f-istiod-ca-cert" pod "d54282f6-acb0-4f9d-9f59-2888caddb88f" (UID: "d54282f6-acb0-4f9d-9f59-2888caddb88f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:33.073648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.073534    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54282f6-acb0-4f9d-9f59-2888caddb88f-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.073648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.073560    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.073648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.073574    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.073648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.073588    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.073648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.073601    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.145661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.117100    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-envoy" pod "d54282f6-acb0-4f9d-9f59-2888caddb88f" (UID: "d54282f6-acb0-4f9d-9f59-2888caddb88f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:33.145661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.124548    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-token" pod "d54282f6-acb0-4f9d-9f59-2888caddb88f" (UID: "d54282f6-acb0-4f9d-9f59-2888caddb88f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:33.145661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.126765    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-podinfo" pod "d54282f6-acb0-4f9d-9f59-2888caddb88f" (UID: "d54282f6-acb0-4f9d-9f59-2888caddb88f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:33.145661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.137868    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-kube-api-access-v2klk" pod "d54282f6-acb0-4f9d-9f59-2888caddb88f" (UID: "d54282f6-acb0-4f9d-9f59-2888caddb88f"). InnerVolumeSpecName "kube-api-access-v2klk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:33.176920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.175721    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.176920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.175756    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.176920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.175773    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2klk\" (UniqueName: \"kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-kube-api-access-v2klk\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.176920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.175789    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54282f6-acb0-4f9d-9f59-2888caddb88f-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:33.270872 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.270424    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc" event={"ID":"b48e7dc6-2291-4349-84a1-40c6aa46a6df","Type":"ContainerStarted","Data":"85c64e08a0cbfabe224fc2862206707f1d12ad846603b6d57ec6a6cc03e9d8ef"}
Oct 29 22:10:33.281813 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.280667    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" event={"ID":"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5","Type":"ContainerStarted","Data":"f2d97a0488c0ff0bd2ec890e1c6a6df336bd08ee69dff90761c1164db9040044"}
Oct 29 22:10:33.282474 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.282027    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"]
Oct 29 22:10:33.284075 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.284052    7082 cpu_manager.go:414] "containerMap: removing container" podUID="d54282f6-acb0-4f9d-9f59-2888caddb88f" containerName="istio-proxy"
Oct 29 22:10:33.284212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.284197    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54282f6-acb0-4f9d-9f59-2888caddb88f" containerName="istio-proxy"
Oct 29 22:10:33.284323 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.284311    7082 cpu_manager.go:414] "containerMap: removing container" podUID="1fa3614e-71e6-4d35-9046-8876a080c622" containerName="istio-proxy"
Oct 29 22:10:33.284408 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.284397    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa3614e-71e6-4d35-9046-8876a080c622" containerName="istio-proxy"
Oct 29 22:10:33.291652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.284877    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54282f6-acb0-4f9d-9f59-2888caddb88f" containerName="istio-proxy"
Oct 29 22:10:33.291652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.284906    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa3614e-71e6-4d35-9046-8876a080c622" containerName="istio-proxy"
Oct 29 22:10:33.291652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.291542    7082 generic.go:356] "Generic (PLEG): container finished" podID="d54282f6-acb0-4f9d-9f59-2888caddb88f" containerID="87ebad69e4423d3dc1a0ebf93e8a1309aa3fb614af9a44879c5657a26ea861bf" exitCode=0
Oct 29 22:10:33.291926 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.291752    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"
Oct 29 22:10:33.370191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.369155    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"]
Oct 29 22:10:33.370191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.369200    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" event={"ID":"4301312f-6325-4e8c-9df7-8e4153fbcea9","Type":"ContainerStarted","Data":"0e6fc54baaf79a5ddfd3a4086a62d24c0a23a388bae038e8665579a235bcd021"}
Oct 29 22:10:33.370191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.369233    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn" event={"ID":"d54282f6-acb0-4f9d-9f59-2888caddb88f","Type":"ContainerDied","Data":"87ebad69e4423d3dc1a0ebf93e8a1309aa3fb614af9a44879c5657a26ea861bf"}
Oct 29 22:10:33.370191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.369252    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn" event={"ID":"d54282f6-acb0-4f9d-9f59-2888caddb88f","Type":"ContainerDied","Data":"99b6bb2c62fdd4fde18cfb09d53b5222767f3720af7cf70efea28349a427d865"}
Oct 29 22:10:33.370191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.369275    7082 scope.go:122] "RemoveContainer" containerID="87ebad69e4423d3dc1a0ebf93e8a1309aa3fb614af9a44879c5657a26ea861bf"
Oct 29 22:10:33.370191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.369484    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.370604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.370534    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" podStartSLOduration=14.370514713 podStartE2EDuration="14.370514713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:33.333845291 +0000 UTC m=+1405.833887590" watchObservedRunningTime="2026-10-29 22:10:33.370514713 +0000 UTC m=+1405.870557012"
Oct 29 22:10:33.384440 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.381985    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"]
Oct 29 22:10:33.433715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.433652    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"]
Oct 29 22:10:33.441327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.441296    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-ggkgn"]
Oct 29 22:10:33.472622 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.463446    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 29 22:10:33.472622 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.463505    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 29 22:10:33.538679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.483901    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3833557-dcfc-4ed5-8415-cca71bba35e7-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.538679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.484105    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.538679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.484168    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.538679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.484297    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.538679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.484345    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.538679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.484572    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.538679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.484611    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcbm7\" (UniqueName: \"kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-kube-api-access-zcbm7\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.538679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.484777    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.538679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.484810    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.588930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.587450    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3833557-dcfc-4ed5-8415-cca71bba35e7-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.588930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.587582    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.588930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.587657    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.588930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.587764    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.588930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.587810    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.588930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.587969    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.588930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.588001    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcbm7\" (UniqueName: \"kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-kube-api-access-zcbm7\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.588930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.588106    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.588930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.588133    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.591770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.590424    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.591770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.590686    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.591770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.590909    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.591770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.591724    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.592415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.592372    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3833557-dcfc-4ed5-8415-cca71bba35e7-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.958787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.945024    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:33.958787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.945277    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:34.024693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:33.997025    7082 scope.go:122] "RemoveContainer" containerID="87ebad69e4423d3dc1a0ebf93e8a1309aa3fb614af9a44879c5657a26ea861bf"
Oct 29 22:10:34.050005 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:10:34.037260    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ebad69e4423d3dc1a0ebf93e8a1309aa3fb614af9a44879c5657a26ea861bf\": container with ID starting with 87ebad69e4423d3dc1a0ebf93e8a1309aa3fb614af9a44879c5657a26ea861bf not found: ID does not exist" containerID="87ebad69e4423d3dc1a0ebf93e8a1309aa3fb614af9a44879c5657a26ea861bf"
Oct 29 22:10:34.050005 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:34.037325    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ebad69e4423d3dc1a0ebf93e8a1309aa3fb614af9a44879c5657a26ea861bf"} err="failed to get container status \"87ebad69e4423d3dc1a0ebf93e8a1309aa3fb614af9a44879c5657a26ea861bf\": rpc error: code = NotFound desc = could not find container \"87ebad69e4423d3dc1a0ebf93e8a1309aa3fb614af9a44879c5657a26ea861bf\": container with ID starting with 87ebad69e4423d3dc1a0ebf93e8a1309aa3fb614af9a44879c5657a26ea861bf not found: ID does not exist"
Oct 29 22:10:34.525706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:34.500410    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcbm7\" (UniqueName: \"kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-kube-api-access-zcbm7\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:34.687057 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:34.669049    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:34.741656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:34.730873    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:10:34.741656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:34.730948    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" podUID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:10:34.838651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:34.797068    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 29 22:10:34.838651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:34.797146    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 29 22:10:35.005652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:34.994256    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:10:35.047993 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.047892    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:35.103139 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.067429    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded"
Oct 29 22:10:35.143301 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.084608    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:35.143902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.126421    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:35.216660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.208112    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:35.414653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.389293    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:10:35.495667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.476486    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" podUID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:10:35.571052 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.571015    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa3614e-71e6-4d35-9046-8876a080c622" path="/var/lib/kubelet/pods/1fa3614e-71e6-4d35-9046-8876a080c622/volumes"
Oct 29 22:10:35.572154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.572100    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3d8a1a-2b77-42cc-8726-bcda92ec29c1" path="/var/lib/kubelet/pods/5b3d8a1a-2b77-42cc-8726-bcda92ec29c1/volumes"
Oct 29 22:10:35.577815 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.577710    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d54282f6-acb0-4f9d-9f59-2888caddb88f" path="/var/lib/kubelet/pods/d54282f6-acb0-4f9d-9f59-2888caddb88f/volumes"
Oct 29 22:10:35.578274 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:10:35.578250    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.623s"
Oct 29 22:10:35.641688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.617673    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:35.641688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.617759    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc" event={"ID":"b48e7dc6-2291-4349-84a1-40c6aa46a6df","Type":"ContainerStarted","Data":"8ebdd95e16ec7b6bef435eb4cde4d2a0cf8a1920956fbe8d9872efe7dbe53310"}
Oct 29 22:10:35.641688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.617820    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"]
Oct 29 22:10:35.641688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.637833    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"
Oct 29 22:10:35.739773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.717906    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 29 22:10:35.739773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.717970    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 29 22:10:35.739773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.736876    7082 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod5f7332e1-ce42-4071-bf1a-ddf706996684"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod5f7332e1-ce42-4071-bf1a-ddf706996684] : Timed out while waiting for systemd to remove kubepods-burstable-pod5f7332e1_ce42_4071_bf1a_ddf706996684.slice"
Oct 29 22:10:35.739773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.736911    7082 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="5f7332e1-ce42-4071-bf1a-ddf706996684" err="failed to delete cgroup paths for [kubepods burstable pod5f7332e1-ce42-4071-bf1a-ddf706996684] : unable to destroy cgroup paths for cgroup [kubepods burstable pod5f7332e1-ce42-4071-bf1a-ddf706996684] : Timed out while waiting for systemd to remove kubepods-burstable-pod5f7332e1_ce42_4071_bf1a_ddf706996684.slice"
Oct 29 22:10:35.769719 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.769684    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a9e67f\""
Oct 29 22:10:36.153084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:35.942281    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:36.153084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.023985    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnwp9\" (UniqueName: \"kubernetes.io/projected/b1a26227-4e48-4b02-8b28-d4eaadc78c80-kube-api-access-jnwp9\") pod \"payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7\" (UID: \"b1a26227-4e48-4b02-8b28-d4eaadc78c80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"
Oct 29 22:10:36.153084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.024445    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a26227-4e48-4b02-8b28-d4eaadc78c80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7\" (UID: \"b1a26227-4e48-4b02-8b28-d4eaadc78c80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"
Oct 29 22:10:36.153084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.101947    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:36.345654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.310865    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwp9\" (UniqueName: \"kubernetes.io/projected/b1a26227-4e48-4b02-8b28-d4eaadc78c80-kube-api-access-jnwp9\") pod \"payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7\" (UID: \"b1a26227-4e48-4b02-8b28-d4eaadc78c80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"
Oct 29 22:10:36.345654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.311289    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a26227-4e48-4b02-8b28-d4eaadc78c80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7\" (UID: \"b1a26227-4e48-4b02-8b28-d4eaadc78c80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"
Oct 29 22:10:36.401878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.401831    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:10:36.507722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.502922    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" podUID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:10:36.507722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.501763    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:36.632659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.577511    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwp9\" (UniqueName: \"kubernetes.io/projected/b1a26227-4e48-4b02-8b28-d4eaadc78c80-kube-api-access-jnwp9\") pod \"payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7\" (UID: \"b1a26227-4e48-4b02-8b28-d4eaadc78c80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"
Oct 29 22:10:36.632659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.615686    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"]
Oct 29 22:10:36.632659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.615852    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:10:36.632659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.615879    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"]
Oct 29 22:10:36.655544 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.655506    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a26227-4e48-4b02-8b28-d4eaadc78c80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7\" (UID: \"b1a26227-4e48-4b02-8b28-d4eaadc78c80\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"
Oct 29 22:10:36.681793 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.681757    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 29 22:10:36.682065 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.682033    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 29 22:10:36.988665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.972843    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"
Oct 29 22:10:36.988665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.985136    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:36.988665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.985537    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:10:36.988665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:36.986563    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:10:37.058657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.024972    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"]
Oct 29 22:10:37.060478 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.060448    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"
Oct 29 22:10:37.191220 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.163472    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:37.266434 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.266400    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a9e67f-dockercfg-ddvxt\""
Oct 29 22:10:37.482272 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.282048    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhzhj\" (UniqueName: \"kubernetes.io/projected/def131f2-c87a-4d18-9ad3-144bfc1d7f67-kube-api-access-jhzhj\") pod \"payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv\" (UID: \"def131f2-c87a-4d18-9ad3-144bfc1d7f67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"
Oct 29 22:10:37.482272 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.282273    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def131f2-c87a-4d18-9ad3-144bfc1d7f67-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv\" (UID: \"def131f2-c87a-4d18-9ad3-144bfc1d7f67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"
Oct 29 22:10:37.482272 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.368330    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"]
Oct 29 22:10:37.482272 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.368483    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.482272 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.386542    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"]
Oct 29 22:10:37.482272 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:10:37.445876    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3833557_dcfc_4ed5_8415_cca71bba35e7.slice/crio-425040e89298e01aac35a9cf382c6a42e7f009bed4214f8705a0df092cfa4818 WatchSource:0}: Error finding container 425040e89298e01aac35a9cf382c6a42e7f009bed4214f8705a0df092cfa4818: Status 404 returned error can't find the container with id 425040e89298e01aac35a9cf382c6a42e7f009bed4214f8705a0df092cfa4818
Oct 29 22:10:37.482272 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.453271    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:10:37.482272 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.453358    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" podUID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:10:37.494981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.484654    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:37.494981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.493173    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 29 22:10:37.494981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.493234    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 29 22:10:37.584306 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.566705    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:10:37.584306 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.567750    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:10:37.659412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.657931    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:37.670561 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.670528    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.670903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.670882    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhzhj\" (UniqueName: \"kubernetes.io/projected/def131f2-c87a-4d18-9ad3-144bfc1d7f67-kube-api-access-jhzhj\") pod \"payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv\" (UID: \"def131f2-c87a-4d18-9ad3-144bfc1d7f67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"
Oct 29 22:10:37.671103 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.671065    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m84t\" (UniqueName: \"kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-kube-api-access-8m84t\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.671389 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.671370    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.673651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.671563    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.673903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.673883    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.677641 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.674202    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def131f2-c87a-4d18-9ad3-144bfc1d7f67-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv\" (UID: \"def131f2-c87a-4d18-9ad3-144bfc1d7f67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"
Oct 29 22:10:37.678164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.678129    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.678392 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.678374    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.825505 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.693756    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.825505 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.693894    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.825505 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.678989    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def131f2-c87a-4d18-9ad3-144bfc1d7f67-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv\" (UID: \"def131f2-c87a-4d18-9ad3-144bfc1d7f67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"
Oct 29 22:10:37.838876 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.838774    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:37.871650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.867091    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:37.871650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.867184    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:37.958297 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.958258    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:10:37.958834 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.958799    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:10:37.959030 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.959018    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:10:37.959152 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.959140    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:10:37.977849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.974351    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.977849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.974399    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.977849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.974530    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.977849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.974974    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:37.977849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.975114    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:37.977849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.975199    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:37.977849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.975254    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.977849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.975495    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.977849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.975679    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.978918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.977948    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.978918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.978189    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.980145 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.980122    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:37.981104 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.979970    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m84t\" (UniqueName: \"kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-kube-api-access-8m84t\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:38.074004 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.071111    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:38.074004 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.071164    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:38.074004 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.071208    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:38.076589 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.075995    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:38.076589 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:37.990897    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:38.082685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.082360    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:38.082685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.067614    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:38.083081 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.083004    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:38.083081 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.071028    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m84t\" (UniqueName: \"kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-kube-api-access-8m84t\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:38.083397 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.083333    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:10:38.180804 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.180414    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhzhj\" (UniqueName: \"kubernetes.io/projected/def131f2-c87a-4d18-9ad3-144bfc1d7f67-kube-api-access-jhzhj\") pod \"payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv\" (UID: \"def131f2-c87a-4d18-9ad3-144bfc1d7f67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"
Oct 29 22:10:38.264673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.249999    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:10:38.264673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.250059    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" podUID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:10:38.264673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.254405    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"]
Oct 29 22:10:38.272618 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.272584    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:10:38.325127 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.296402    7082 generic.go:356] "Generic (PLEG): container finished" podID="b48e7dc6-2291-4349-84a1-40c6aa46a6df" containerID="8ebdd95e16ec7b6bef435eb4cde4d2a0cf8a1920956fbe8d9872efe7dbe53310" exitCode=0
Oct 29 22:10:38.325127 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.302935    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc" event={"ID":"b48e7dc6-2291-4349-84a1-40c6aa46a6df","Type":"ContainerDied","Data":"8ebdd95e16ec7b6bef435eb4cde4d2a0cf8a1920956fbe8d9872efe7dbe53310"}
Oct 29 22:10:38.377212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.376911    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"
Oct 29 22:10:38.381750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.381474    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:38.474140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.470402    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 29 22:10:38.474140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.470474    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 29 22:10:38.617789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:38.617709    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:39.276811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.246156    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:10:39.276811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.246216    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" podUID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:10:39.366664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.356177    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:39.366664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.356246    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:39.472800 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.472761    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 29 22:10:39.474055 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.472825    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 29 22:10:39.766652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.731590    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:39.922706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.922166    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istiod-ca-cert\") pod \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") "
Oct 29 22:10:39.922706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.922298    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-kube-api-access-f5xpd\" (UniqueName: \"kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-kube-api-access-f5xpd\") pod \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") "
Oct 29 22:10:39.922706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.922397    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-envoy\") pod \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") "
Oct 29 22:10:39.922706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.922518    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-podinfo\") pod \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") "
Oct 29 22:10:39.922706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.922601    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-data\") pod \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") "
Oct 29 22:10:39.923122 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.922776    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-certs\") pod \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") "
Oct 29 22:10:39.923122 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.922814    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-credential-socket\") pod \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") "
Oct 29 22:10:39.957940 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.957687    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-socket\") pod \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") "
Oct 29 22:10:39.957940 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.957824    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-token\" (UniqueName: \"kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-token\") pod \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\" (UID: \"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85\") "
Oct 29 22:10:39.989216 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.989170    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-data" pod "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" (UID: "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.989466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.989443    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-certs" pod "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" (UID: "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.994650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.990339    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-credential-socket" pod "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" (UID: "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.994650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.990657    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-socket" pod "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" (UID: "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.998184 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.998159    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-podinfo" pod "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" (UID: "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:39.998618 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:39.998596    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istiod-ca-cert" pod "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" (UID: "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:40.001497 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.001244    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-kube-api-access-f5xpd" pod "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" (UID: "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85"). InnerVolumeSpecName "kube-api-access-f5xpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:40.140237 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.098217    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.201274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.191435    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.201274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.191466    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.201274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.191484    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.201274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.191499    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5xpd\" (UniqueName: \"kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-kube-api-access-f5xpd\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.201274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.191514    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.201274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.191528    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.201274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.142426    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-token" pod "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" (UID: "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:40.266923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.265952    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:10:40.266923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.266015    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" podUID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:10:40.296409 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.296364    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.372320 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.372288    7082 generic.go:356] "Generic (PLEG): container finished" podID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerID="8ea8f72cc4dfa211d66730ae265552c355d8e2a4f2707b27b3f93e555afcd216" exitCode=0
Oct 29 22:10:40.372663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.372615    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" event={"ID":"e6fc8d32-632e-4ff3-90cb-f80fde70f134","Type":"ContainerDied","Data":"8ea8f72cc4dfa211d66730ae265552c355d8e2a4f2707b27b3f93e555afcd216"}
Oct 29 22:10:40.373966 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.373928    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"]
Oct 29 22:10:40.375964 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.375942    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:40.378074 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.378046    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" event={"ID":"d3833557-dcfc-4ed5-8415-cca71bba35e7","Type":"ContainerStarted","Data":"425040e89298e01aac35a9cf382c6a42e7f009bed4214f8705a0df092cfa4818"}
Oct 29 22:10:40.379438 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.379422    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"
Oct 29 22:10:40.384541 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.382027    7082 generic.go:356] "Generic (PLEG): container finished" podID="76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" containerID="1c229e641dd1e9f76e91dbe9df7ad745e8d3e22243f4951bc86ca738618701a9" exitCode=0
Oct 29 22:10:40.384541 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.383209    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"
Oct 29 22:10:40.384541 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.383367    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d" event={"ID":"76ec3d3b-6665-4e50-8e47-8c7e7b09ba85","Type":"ContainerDied","Data":"1c229e641dd1e9f76e91dbe9df7ad745e8d3e22243f4951bc86ca738618701a9"}
Oct 29 22:10:40.384541 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.383403    7082 scope.go:122] "RemoveContainer" containerID="1c229e641dd1e9f76e91dbe9df7ad745e8d3e22243f4951bc86ca738618701a9"
Oct 29 22:10:40.426661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.424791    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:40.427655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.427589    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-envoy" pod "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" (UID: "76ec3d3b-6665-4e50-8e47-8c7e7b09ba85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:40.476699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.475983    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 29 22:10:40.476699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.476051    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.513413    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-certs\") pod \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") "
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.513469    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-credential-socket\") pod \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") "
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.513534    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-data\") pod \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") "
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.513589    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-kube-api-access-vf9j4\" (UniqueName: \"kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-kube-api-access-vf9j4\") pod \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") "
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.513648    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-openshift-ca\") pod \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\" (UID: \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\") "
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.513799    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-xks-ca\") pod \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\" (UID: \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\") "
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.513913    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-envoy\") pod \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") "
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.513981    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istiod-ca-cert\") pod \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") "
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.514023    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-token\") pod \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") "
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.514198    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-podinfo\") pod \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") "
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.514280    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-socket\") pod \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\" (UID: \"e6fc8d32-632e-4ff3-90cb-f80fde70f134\") "
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.514330    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48e7dc6-2291-4349-84a1-40c6aa46a6df-kube-api-access-j4nbm\" (UniqueName: \"kubernetes.io/projected/b48e7dc6-2291-4349-84a1-40c6aa46a6df-kube-api-access-j4nbm\") pod \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\" (UID: \"b48e7dc6-2291-4349-84a1-40c6aa46a6df\") "
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.515512    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.516140    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-certs" pod "e6fc8d32-632e-4ff3-90cb-f80fde70f134" (UID: "e6fc8d32-632e-4ff3-90cb-f80fde70f134"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.516425    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-credential-socket" pod "e6fc8d32-632e-4ff3-90cb-f80fde70f134" (UID: "e6fc8d32-632e-4ff3-90cb-f80fde70f134"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:40.538766 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.538302    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-data" pod "e6fc8d32-632e-4ff3-90cb-f80fde70f134" (UID: "e6fc8d32-632e-4ff3-90cb-f80fde70f134"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:40.558975 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.558925    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:40.637855 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.637480    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istiod-ca-cert" pod "e6fc8d32-632e-4ff3-90cb-f80fde70f134" (UID: "e6fc8d32-632e-4ff3-90cb-f80fde70f134"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:40.650933 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.650812    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48e7dc6-2291-4349-84a1-40c6aa46a6df-kube-api-access-j4nbm" pod "b48e7dc6-2291-4349-84a1-40c6aa46a6df" (UID: "b48e7dc6-2291-4349-84a1-40c6aa46a6df"). InnerVolumeSpecName "kube-api-access-j4nbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:40.666275 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.664436    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4nbm\" (UniqueName: \"kubernetes.io/projected/b48e7dc6-2291-4349-84a1-40c6aa46a6df-kube-api-access-j4nbm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.666275 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.664468    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.666275 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.664483    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.666275 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.664499    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.666275 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.664514    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.681972 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.681840    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-openshift-ca" pod "b48e7dc6-2291-4349-84a1-40c6aa46a6df" (UID: "b48e7dc6-2291-4349-84a1-40c6aa46a6df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:40.685427 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.685391    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-socket" pod "e6fc8d32-632e-4ff3-90cb-f80fde70f134" (UID: "e6fc8d32-632e-4ff3-90cb-f80fde70f134"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:40.718489 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.718387    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" podStartSLOduration=7.718366157 podStartE2EDuration="7.718366157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:40.690562363 +0000 UTC m=+1413.190604662" watchObservedRunningTime="2026-10-29 22:10:40.718366157 +0000 UTC m=+1413.218408457"
Oct 29 22:10:40.818652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.814180    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.818652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.814223    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.831295 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.830726    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-token" pod "e6fc8d32-632e-4ff3-90cb-f80fde70f134" (UID: "e6fc8d32-632e-4ff3-90cb-f80fde70f134"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:40.833013 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.832817    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-kube-api-access-vf9j4" pod "e6fc8d32-632e-4ff3-90cb-f80fde70f134" (UID: "e6fc8d32-632e-4ff3-90cb-f80fde70f134"). InnerVolumeSpecName "kube-api-access-vf9j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:40.833013 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.832901    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-podinfo" pod "e6fc8d32-632e-4ff3-90cb-f80fde70f134" (UID: "e6fc8d32-632e-4ff3-90cb-f80fde70f134"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:40.833013 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.832957    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-xks-ca" pod "b48e7dc6-2291-4349-84a1-40c6aa46a6df" (UID: "b48e7dc6-2291-4349-84a1-40c6aa46a6df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:40.834812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.834770    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-envoy" pod "e6fc8d32-632e-4ff3-90cb-f80fde70f134" (UID: "e6fc8d32-632e-4ff3-90cb-f80fde70f134"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:40.928576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.925689    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48e7dc6-2291-4349-84a1-40c6aa46a6df-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.928576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.925730    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.928576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.925744    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.928576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.925760    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6fc8d32-632e-4ff3-90cb-f80fde70f134-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:40.928576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:40.925772    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf9j4\" (UniqueName: \"kubernetes.io/projected/e6fc8d32-632e-4ff3-90cb-f80fde70f134-kube-api-access-vf9j4\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:41.104555 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.103160    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:41.207114 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:10:41.202730    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb3aa0d_adb0_4e8f_8a8b_1e184e0be714.slice/crio-37af08d97cc7e8b81af7b769895b73867f48176cee9c51143276e21826a7240c WatchSource:0}: Error finding container 37af08d97cc7e8b81af7b769895b73867f48176cee9c51143276e21826a7240c: Status 404 returned error can't find the container with id 37af08d97cc7e8b81af7b769895b73867f48176cee9c51143276e21826a7240c
Oct 29 22:10:41.266866 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.266829    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 22:10:41.273656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.267125    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 22:10:41.277789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.277716    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"]
Oct 29 22:10:41.357747 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.357700    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-2949d"]
Oct 29 22:10:41.406811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.403673    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"]
Oct 29 22:10:41.406811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.403838    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:41.406811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.405525    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:41.407255 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.407199    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:10:41.453420 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.452208    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:41.453420 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.452312    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:41.453420 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.452353    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:41.528534 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.528493    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"]
Oct 29 22:10:41.528820 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.528804    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"]
Oct 29 22:10:41.537673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.535758    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:41.567109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.567018    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv" event={"ID":"def131f2-c87a-4d18-9ad3-144bfc1d7f67","Type":"ContainerStarted","Data":"04efead485f224ee14138d71c07c15fda834a115615e65cc712ef2338721dfbd"}
Oct 29 22:10:41.569831 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.569726    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" event={"ID":"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714","Type":"ContainerStarted","Data":"37af08d97cc7e8b81af7b769895b73867f48176cee9c51143276e21826a7240c"}
Oct 29 22:10:41.577450 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.574085    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7" event={"ID":"b1a26227-4e48-4b02-8b28-d4eaadc78c80","Type":"ContainerStarted","Data":"0760a8dcae0649dcbc31a4124f6007d9b95b5805e426c348e2908b36eacfccb5"}
Oct 29 22:10:41.577450 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.574146    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7" event={"ID":"b1a26227-4e48-4b02-8b28-d4eaadc78c80","Type":"ContainerStarted","Data":"98ceaa080c85937979dccb4e437fcdd95a51a8161458c5f36e01cb58814ff110"}
Oct 29 22:10:41.577450 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.575233    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"
Oct 29 22:10:41.580985 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.580964    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"
Oct 29 22:10:41.581519 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.581020    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg" event={"ID":"e6fc8d32-632e-4ff3-90cb-f80fde70f134","Type":"ContainerDied","Data":"07a5dfb8f04b7d6b66690be8533c57709697e02d38b9d6f52c0aea92fa66ddb1"}
Oct 29 22:10:41.581519 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.581275    7082 scope.go:122] "RemoveContainer" containerID="8ea8f72cc4dfa211d66730ae265552c355d8e2a4f2707b27b3f93e555afcd216"
Oct 29 22:10:41.589687 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.589583    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" event={"ID":"d3833557-dcfc-4ed5-8415-cca71bba35e7","Type":"ContainerStarted","Data":"27e45bf0b05cea3c60d77a9b6d0b2b07080776303d0407c3630995d67f70b061"}
Oct 29 22:10:41.595533 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.594234    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"
Oct 29 22:10:41.597454 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.596454    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc" event={"ID":"b48e7dc6-2291-4349-84a1-40c6aa46a6df","Type":"ContainerDied","Data":"85c64e08a0cbfabe224fc2862206707f1d12ad846603b6d57ec6a6cc03e9d8ef"}
Oct 29 22:10:41.597454 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.596518    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85c64e08a0cbfabe224fc2862206707f1d12ad846603b6d57ec6a6cc03e9d8ef"
Oct 29 22:10:41.602383 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.602051    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:10:41.870834 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.870753    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" path="/var/lib/kubelet/pods/76ec3d3b-6665-4e50-8e47-8c7e7b09ba85/volumes"
Oct 29 22:10:41.876102 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.876038    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7" podStartSLOduration=7.876012691 podStartE2EDuration="7.876012691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:41.867195074 +0000 UTC m=+1414.367237372" watchObservedRunningTime="2026-10-29 22:10:41.876012691 +0000 UTC m=+1414.376054986"
Oct 29 22:10:41.964199 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.963495    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"]
Oct 29 22:10:41.976962 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:41.976921    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-n66pg"]
Oct 29 22:10:42.134696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:42.125522    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 22:10:42.135191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:42.135128    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 22:10:42.572657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:42.564780    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:42.572657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:42.564838    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:42.647763 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:42.647722    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv" event={"ID":"def131f2-c87a-4d18-9ad3-144bfc1d7f67","Type":"ContainerStarted","Data":"198d2a831be01ae3b0504c3cca592fae36c71a817541fae58b9a6d793078c663"}
Oct 29 22:10:42.650079 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:42.650056    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"
Oct 29 22:10:42.756653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:42.724883    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" event={"ID":"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714","Type":"ContainerStarted","Data":"4ef73f3dcfed5824f9c9e0a619a584045688182a941e877a323638046f068db5"}
Oct 29 22:10:42.912577 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:42.912449    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" podStartSLOduration=8.912427935 podStartE2EDuration="8.912427935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:42.841175273 +0000 UTC m=+1415.341217578" watchObservedRunningTime="2026-10-29 22:10:42.912427935 +0000 UTC m=+1415.412470237"
Oct 29 22:10:42.974661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:42.924692    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv" podStartSLOduration=8.924673268 podStartE2EDuration="8.924673268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:42.72589454 +0000 UTC m=+1415.225936839" watchObservedRunningTime="2026-10-29 22:10:42.924673268 +0000 UTC m=+1415.424715568"
Oct 29 22:10:43.336652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:43.115619    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 22:10:43.336652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:43.336046    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 22:10:43.384477 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:43.382256    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:43.384477 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:43.384158    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 29 22:10:43.384477 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:43.384211    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 29 22:10:43.758706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:43.758372    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" path="/var/lib/kubelet/pods/e6fc8d32-632e-4ff3-90cb-f80fde70f134/volumes"
Oct 29 22:10:44.073691 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.072615    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"]
Oct 29 22:10:44.073906 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.073846    7082 cpu_manager.go:414] "containerMap: removing container" podUID="b48e7dc6-2291-4349-84a1-40c6aa46a6df" containerName="revoke-keys"
Oct 29 22:10:44.073906 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.073864    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48e7dc6-2291-4349-84a1-40c6aa46a6df" containerName="revoke-keys"
Oct 29 22:10:44.073906 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.073895    7082 cpu_manager.go:414] "containerMap: removing container" podUID="76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" containerName="istio-proxy"
Oct 29 22:10:44.073906 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.073902    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" containerName="istio-proxy"
Oct 29 22:10:44.074121 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.073932    7082 cpu_manager.go:414] "containerMap: removing container" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy"
Oct 29 22:10:44.074121 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.073940    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy"
Oct 29 22:10:44.074228 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.074126    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ec3d3b-6665-4e50-8e47-8c7e7b09ba85" containerName="istio-proxy"
Oct 29 22:10:44.074228 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.074143    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6fc8d32-632e-4ff3-90cb-f80fde70f134" containerName="istio-proxy"
Oct 29 22:10:44.074228 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.074164    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48e7dc6-2291-4349-84a1-40c6aa46a6df" containerName="revoke-keys"
Oct 29 22:10:44.108652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.089978    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 22:10:44.108652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.090039    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 22:10:44.157650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.151673    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"]
Oct 29 22:10:44.194942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.186761    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.236663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.236040    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a9e67f-openshift-default-dockercfg-kbmzv\""
Oct 29 22:10:44.279711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.279445    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"]
Oct 29 22:10:44.279711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.279494    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"]
Oct 29 22:10:44.279711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.279551    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.363901 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.340050    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" containerID="cri-o://70664f7b0631ef941b485d77c1a48c04119bd96fbee915ba3a22a387a5b7cabb" gracePeriod=34
Oct 29 22:10:44.382759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.382444    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 29 22:10:44.382759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.382486    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 29 22:10:44.383569 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.383544    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.383656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.383584    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grzq6\" (UniqueName: \"kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-kube-api-access-grzq6\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.383813 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.383784    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.383868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.383833    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.383956 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.383926    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.384111 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.383981    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.384323 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.384104    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.384323 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.384151    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.384323 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.384245    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee700b95-a323-410b-8fc8-1e5198dcf771-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.384323 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.384277    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.384323 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.384322    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp6pb\" (UniqueName: \"kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-kube-api-access-lp6pb\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.384684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.384370    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.384684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.384396    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.384684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.384444    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.384684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.384461    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.384684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.384485    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1116dd4-7fad-45d8-8508-f0283b431cfa-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.384684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.384588    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.384684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.384601    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.486473    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.486528    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grzq6\" (UniqueName: \"kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-kube-api-access-grzq6\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.486599    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.486643    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.486695    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.486720    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.486788    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.486821    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.486886    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee700b95-a323-410b-8fc8-1e5198dcf771-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.486914    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.486970    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp6pb\" (UniqueName: \"kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-kube-api-access-lp6pb\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.487031    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.487066    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.487131    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.487155    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.491047    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.502762    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1116dd4-7fad-45d8-8508-f0283b431cfa-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.502996    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.503028    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.503556    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.505316 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.504616    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.520909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.514921    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.520909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.516613    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee700b95-a323-410b-8fc8-1e5198dcf771-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.520909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.516880    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.547120 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.534781    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1116dd4-7fad-45d8-8508-f0283b431cfa-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.547723 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.547641    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.548225 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.548124    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.548621 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.548570    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.549423 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.549367    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.550315 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.550289    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.551968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.551920    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.552142 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.552126    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.552778 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.552736    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.554818 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.554801    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grzq6\" (UniqueName: \"kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-kube-api-access-grzq6\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.567165 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.567132    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.567311 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.567262    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp6pb\" (UniqueName: \"kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-kube-api-access-lp6pb\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.623470 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.623284    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": dial tcp 10.134.0.96:8000: connect: connection refused"
Oct 29 22:10:44.756502 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.756473    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:44.835980 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.834921    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:44.946357 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.946321    7082 generic.go:356] "Generic (PLEG): container finished" podID="9900ad31-588e-4077-bef3-1ee57af99f84" containerID="70664f7b0631ef941b485d77c1a48c04119bd96fbee915ba3a22a387a5b7cabb" exitCode=0
Oct 29 22:10:44.949688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:44.946591    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" event={"ID":"9900ad31-588e-4077-bef3-1ee57af99f84","Type":"ContainerDied","Data":"70664f7b0631ef941b485d77c1a48c04119bd96fbee915ba3a22a387a5b7cabb"}
Oct 29 22:10:45.134819 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.093136    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 22:10:45.134819 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.093195    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 22:10:45.504929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.445021    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 29 22:10:45.504929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.445085    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 29 22:10:45.688957 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.683810    7082 scope.go:122] "RemoveContainer" containerID="b502183142e9f7874b53527e908cfb6d7448b9fcf58eb11b0195e74c3aef5dc1"
Oct 29 22:10:45.774487 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.774463    7082 scope.go:122] "RemoveContainer" containerID="bbcd32d0c757872de42d79d2f1901f5adc21a804e980e24155106073a784b501"
Oct 29 22:10:45.776618 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.776588    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:10:45.865321 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.863397    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9900ad31-588e-4077-bef3-1ee57af99f84-kube-api-access-7tfhv\" (UniqueName: \"kubernetes.io/projected/9900ad31-588e-4077-bef3-1ee57af99f84-kube-api-access-7tfhv\") pod \"9900ad31-588e-4077-bef3-1ee57af99f84\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") "
Oct 29 22:10:45.865321 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.863493    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-model-cache\") pod \"9900ad31-588e-4077-bef3-1ee57af99f84\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") "
Oct 29 22:10:45.865321 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.863690    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9900ad31-588e-4077-bef3-1ee57af99f84-tls-certs\" (UniqueName: \"kubernetes.io/secret/9900ad31-588e-4077-bef3-1ee57af99f84-tls-certs\") pod \"9900ad31-588e-4077-bef3-1ee57af99f84\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") "
Oct 29 22:10:45.865321 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.863730    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-dshm\") pod \"9900ad31-588e-4077-bef3-1ee57af99f84\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") "
Oct 29 22:10:45.865321 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.863790    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-home\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-home\") pod \"9900ad31-588e-4077-bef3-1ee57af99f84\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") "
Oct 29 22:10:45.865321 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.864043    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-tmp-dir\") pod \"9900ad31-588e-4077-bef3-1ee57af99f84\" (UID: \"9900ad31-588e-4077-bef3-1ee57af99f84\") "
Oct 29 22:10:45.866049 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.865333    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-home" pod "9900ad31-588e-4077-bef3-1ee57af99f84" (UID: "9900ad31-588e-4077-bef3-1ee57af99f84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:45.866049 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.865482    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-model-cache" pod "9900ad31-588e-4077-bef3-1ee57af99f84" (UID: "9900ad31-588e-4077-bef3-1ee57af99f84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:45.866049 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.865708    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-tmp-dir" pod "9900ad31-588e-4077-bef3-1ee57af99f84" (UID: "9900ad31-588e-4077-bef3-1ee57af99f84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:45.869066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.869029    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9900ad31-588e-4077-bef3-1ee57af99f84-tls-certs" pod "9900ad31-588e-4077-bef3-1ee57af99f84" (UID: "9900ad31-588e-4077-bef3-1ee57af99f84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:45.869614 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.869589    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-dshm" pod "9900ad31-588e-4077-bef3-1ee57af99f84" (UID: "9900ad31-588e-4077-bef3-1ee57af99f84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:45.871882 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.871846    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9900ad31-588e-4077-bef3-1ee57af99f84-kube-api-access-7tfhv" pod "9900ad31-588e-4077-bef3-1ee57af99f84" (UID: "9900ad31-588e-4077-bef3-1ee57af99f84"). InnerVolumeSpecName "kube-api-access-7tfhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:45.981345 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.981236    7082 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:45.981345 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.981269    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tfhv\" (UniqueName: \"kubernetes.io/projected/9900ad31-588e-4077-bef3-1ee57af99f84-kube-api-access-7tfhv\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:45.981345 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.981284    7082 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:45.981345 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.981299    7082 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9900ad31-588e-4077-bef3-1ee57af99f84-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:45.981345 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.981310    7082 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:45.981345 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:45.981321    7082 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9900ad31-588e-4077-bef3-1ee57af99f84-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:46.062712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.057908    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"]
Oct 29 22:10:46.063325 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.063215    7082 scope.go:122] "RemoveContainer" containerID="0bb1d54e8f31c6949786dbbbfb1ac012b410a38ee011d22c599105597d4b62a2"
Oct 29 22:10:46.076215 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:10:46.070595    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1116dd4_7fad_45d8_8508_f0283b431cfa.slice/crio-4605b5f36e43a05bbcaf81eb7519bbdb984c8a2c310fb390c7d5f8dd5e49f883 WatchSource:0}: Error finding container 4605b5f36e43a05bbcaf81eb7519bbdb984c8a2c310fb390c7d5f8dd5e49f883: Status 404 returned error can't find the container with id 4605b5f36e43a05bbcaf81eb7519bbdb984c8a2c310fb390c7d5f8dd5e49f883
Oct 29 22:10:46.076215 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.074076    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:46.076215 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.074175    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:46.076215 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.074219    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:46.116981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.082888    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"
Oct 29 22:10:46.116981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.083079    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" event={"ID":"9900ad31-588e-4077-bef3-1ee57af99f84","Type":"ContainerDied","Data":"eaa2c17d461f24e4ee1557cd2a9acaf1fad8b5098fe71e826867034435f598b9"}
Oct 29 22:10:46.116981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.083131    7082 scope.go:122] "RemoveContainer" containerID="70664f7b0631ef941b485d77c1a48c04119bd96fbee915ba3a22a387a5b7cabb"
Oct 29 22:10:46.149696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.145811    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 22:10:46.149696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.145859    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 22:10:46.152286 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.152256    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"]
Oct 29 22:10:46.167792 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.167332    7082 scope.go:122] "RemoveContainer" containerID="4d61e0996c289fac326d65d84ede8440a913f31bcd909aaada6a19455e87094d"
Oct 29 22:10:46.170192 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:10:46.170131    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee700b95_a323_410b_8fc8_1e5198dcf771.slice/crio-4e1d11dbd0048728c7e7388c549899b8e2753a8ceb041e94d2cd53cc99ed3138 WatchSource:0}: Error finding container 4e1d11dbd0048728c7e7388c549899b8e2753a8ceb041e94d2cd53cc99ed3138: Status 404 returned error can't find the container with id 4e1d11dbd0048728c7e7388c549899b8e2753a8ceb041e94d2cd53cc99ed3138
Oct 29 22:10:46.266385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.265718    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"]
Oct 29 22:10:46.266876 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.266854    7082 scope.go:122] "RemoveContainer" containerID="b57e57d9dc87faf6187a1df1439f82967ecd7beb338a81db6ccba1f4a0db12ed"
Oct 29 22:10:46.268468 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.268430    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:46.268553 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.268520    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:46.268602 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.268566    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:46.274570 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.274541    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z"]
Oct 29 22:10:46.366116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.366096    7082 scope.go:122] "RemoveContainer" containerID="70664f7b0631ef941b485d77c1a48c04119bd96fbee915ba3a22a387a5b7cabb"
Oct 29 22:10:46.366518 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:10:46.366404    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70664f7b0631ef941b485d77c1a48c04119bd96fbee915ba3a22a387a5b7cabb\": container with ID starting with 70664f7b0631ef941b485d77c1a48c04119bd96fbee915ba3a22a387a5b7cabb not found: ID does not exist" containerID="70664f7b0631ef941b485d77c1a48c04119bd96fbee915ba3a22a387a5b7cabb"
Oct 29 22:10:46.366518 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:10:46.366432    7082 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"70664f7b0631ef941b485d77c1a48c04119bd96fbee915ba3a22a387a5b7cabb\": rpc error: code = NotFound desc = could not find container \"70664f7b0631ef941b485d77c1a48c04119bd96fbee915ba3a22a387a5b7cabb\": container with ID starting with 70664f7b0631ef941b485d77c1a48c04119bd96fbee915ba3a22a387a5b7cabb not found: ID does not exist" containerID="70664f7b0631ef941b485d77c1a48c04119bd96fbee915ba3a22a387a5b7cabb"
Oct 29 22:10:46.366518 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.366451    7082 scope.go:122] "RemoveContainer" containerID="9ee41bc1d36c3df38853a8288e0831bba91c5dedc65efe2a737802a52a79c3a1"
Oct 29 22:10:46.383444 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.383412    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 29 22:10:46.383566 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.383480    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 29 22:10:46.522826 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.519844    7082 scope.go:122] "RemoveContainer" containerID="0cf9ffd080269e9a7546f34b822a3e849212925fa3a22032693be47af9d1bfc9"
Oct 29 22:10:46.530666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.529211    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x" podUID="79e8c570-3e0f-40b7-9696-81b84b10e8eb" containerName="payload-processing" containerID="cri-o://e3ab74d6d6c07120f1861fbc0a0b1cea3fe8f7e62a72f6f6ded5d2e329656226" gracePeriod=25
Oct 29 22:10:46.725331 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.701224    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/distinct-w2-a9def7-kserve-77bcbd688f-jpb2z" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:46.932652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:46.910145    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x" podUID="79e8c570-3e0f-40b7-9696-81b84b10e8eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:10:47.111013 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:47.088261    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 22:10:47.111013 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:47.088328    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 22:10:47.350995 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:47.350958    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" event={"ID":"a1116dd4-7fad-45d8-8508-f0283b431cfa","Type":"ContainerStarted","Data":"75b55adc30bf4aa1dd1784476c5d45c4bc3df5775c4061228befc71b0c948d29"}
Oct 29 22:10:47.356222 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:47.351254    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" event={"ID":"a1116dd4-7fad-45d8-8508-f0283b431cfa","Type":"ContainerStarted","Data":"4605b5f36e43a05bbcaf81eb7519bbdb984c8a2c310fb390c7d5f8dd5e49f883"}
Oct 29 22:10:47.387988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:47.387953    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 29 22:10:47.388350 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:47.388317    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 29 22:10:47.475244 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:47.460739    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" event={"ID":"ee700b95-a323-410b-8fc8-1e5198dcf771","Type":"ContainerStarted","Data":"4e1d11dbd0048728c7e7388c549899b8e2753a8ceb041e94d2cd53cc99ed3138"}
Oct 29 22:10:47.530212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:47.518333    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" podStartSLOduration=4.518312153 podStartE2EDuration="4.518312153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:47.44247287 +0000 UTC m=+1419.942515172" watchObservedRunningTime="2026-10-29 22:10:47.518312153 +0000 UTC m=+1420.018354451"
Oct 29 22:10:47.860474 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:47.859982    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 29 22:10:47.860474 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:47.860055    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 29 22:10:47.860803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:47.860597    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" path="/var/lib/kubelet/pods/9900ad31-588e-4077-bef3-1ee57af99f84/volumes"
Oct 29 22:10:47.861194 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:47.861141    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:48.105103 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:48.093039    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 29 22:10:48.105103 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:48.093115    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 29 22:10:48.382946 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:48.382495    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 29 22:10:48.382946 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:48.382550    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 29 22:10:48.640707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:48.632908    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" containerID="cri-o://670529ce9d5bcd94f25e790d10960dc5cbd5ebc47e797d59329c8ea6c9bff676" gracePeriod=45
Oct 29 22:10:48.813699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:48.809807    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" event={"ID":"ee700b95-a323-410b-8fc8-1e5198dcf771","Type":"ContainerStarted","Data":"d6beeb2b3a39d45ff3dbc9c806f0d9e378a344b6af3931440c30135d0aff0a0b"}
Oct 29 22:10:48.854813 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:48.837063    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 29 22:10:48.854813 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:48.837133    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 29 22:10:49.081141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:49.080759    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" podStartSLOduration=6.080717676 podStartE2EDuration="6.080717676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:48.894672174 +0000 UTC m=+1421.394714466" watchObservedRunningTime="2026-10-29 22:10:49.080717676 +0000 UTC m=+1421.580759976"
Oct 29 22:10:49.126757 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:49.118808    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"]
Oct 29 22:10:49.672040 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:49.601452    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 29 22:10:49.672040 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:49.601515    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 29 22:10:49.768106 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:49.733763    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:49.836228 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:49.827949    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded"
Oct 29 22:10:49.901382 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:49.891858    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:10:49.901382 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:49.891978    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" podUID="ee700b95-a323-410b-8fc8-1e5198dcf771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:10:49.901382 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:49.892106    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection reset by peer"
Oct 29 22:10:49.911828 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:49.911752    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:49.989291 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:49.937018    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 29 22:10:49.989291 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:49.949724    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 29 22:10:50.136996 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:50.045727    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:50.262654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:50.225107    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:50.739052 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:50.722141    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:50.770001 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:50.744444    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 29 22:10:50.770001 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:50.746499    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:50.770001 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:10:50.767546    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.127s"
Oct 29 22:10:50.870649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:50.870554    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:50.870926 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:50.870908    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"]
Oct 29 22:10:50.873499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:50.873475    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:50.979659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:50.903677    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:50.979659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:50.903774    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:10:50.979659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:50.903899    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:10:51.083156 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.073031    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 29 22:10:51.083156 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.073090    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 29 22:10:51.083562 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.083533    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 29 22:10:51.083746 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.083721    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 29 22:10:51.083935 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.083909    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:10:51.084059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.084042    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" podUID="ee700b95-a323-410b-8fc8-1e5198dcf771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:10:51.350667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.350538    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:51.381662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.366809    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:10:51.381662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.371908    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:10:51.381662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.373535    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:10:51.532130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.514804    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:10:51.713117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.713029    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 29 22:10:51.734699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.725121    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:51.746987 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.746696    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:51.809651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.808829    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:10:51.809651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.808891    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" podUID="ee700b95-a323-410b-8fc8-1e5198dcf771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:10:51.876649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.862042    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 29 22:10:51.876649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.862107    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 29 22:10:51.876649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.865700    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 29 22:10:51.876649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.865748    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 29 22:10:51.876649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:51.867717    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:10:52.329488 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:52.314965    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:52.419609 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:52.418343    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:52.668701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:52.591177    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:10:52.678815 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:10:52.675400    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2ccb6e6_e9bf_4bb0_9a4e_c27b1ae52f5b.slice/crio-670529ce9d5bcd94f25e790d10960dc5cbd5ebc47e797d59329c8ea6c9bff676.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:10:52.702213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:52.682848    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:10:52.765700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:52.758157    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:10:52.767771 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:52.767482    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:10:52.793377 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:52.793347    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:10:52.793640 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:52.793595    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" podUID="ee700b95-a323-410b-8fc8-1e5198dcf771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:10:52.814957 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:52.796803    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 29 22:10:52.839346 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:52.824327    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:10:52.908359 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:52.908327    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 29 22:10:53.050242 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:53.050211    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 29 22:10:53.269437 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:53.269400    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:53.272537 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:53.272342    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:53.273127 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:53.273026    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:10:53.280423 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:53.280399    7082 generic.go:356] "Generic (PLEG): container finished" podID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerID="670529ce9d5bcd94f25e790d10960dc5cbd5ebc47e797d59329c8ea6c9bff676" exitCode=0
Oct 29 22:10:53.282319 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:53.282289    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" event={"ID":"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b","Type":"ContainerDied","Data":"670529ce9d5bcd94f25e790d10960dc5cbd5ebc47e797d59329c8ea6c9bff676"}
Oct 29 22:10:53.775696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:53.766422    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:10:53.775696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:53.766496    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" podUID="ee700b95-a323-410b-8fc8-1e5198dcf771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:10:53.849439 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:53.847862    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 29 22:10:53.849439 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:53.847919    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 29 22:10:53.879038 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:53.878772    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"
Oct 29 22:10:53.950284 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:53.947768    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:10:54.067655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.067328    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tmp-dir\") pod \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") "
Oct 29 22:10:54.067655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.067472    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tls-certs\") pod \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") "
Oct 29 22:10:54.067655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.067651    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-kube-api-access-h7s4j\" (UniqueName: \"kubernetes.io/projected/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-kube-api-access-h7s4j\") pod \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") "
Oct 29 22:10:54.068000 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.067694    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-home\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-home\") pod \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") "
Oct 29 22:10:54.068000 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.067810    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-dshm\") pod \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") "
Oct 29 22:10:54.068000 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.067945    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-model-cache\") pod \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\" (UID: \"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b\") "
Oct 29 22:10:54.069757 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.069729    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-model-cache" pod "d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" (UID: "d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:54.070384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.070356    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tmp-dir" pod "d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" (UID: "d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:54.072689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.071291    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-home" pod "d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" (UID: "d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:54.072972 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.072934    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tls-certs" pod "d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" (UID: "d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:54.073669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.073382    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-kube-api-access-h7s4j" pod "d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" (UID: "d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b"). InnerVolumeSpecName "kube-api-access-h7s4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:54.160976 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.133988    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb" podUID="7b205272-e4c0-432c-860d-162d9e80f950" containerName="payload-processing" containerID="cri-o://f7cc9d56f53ee9826288e4c70970cd7d3f9f94fec4d1b0011c1aa09bf88f511c" gracePeriod=25
Oct 29 22:10:54.160976 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.137349    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-dshm" pod "d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" (UID: "d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:54.289649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.282153    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7s4j\" (UniqueName: \"kubernetes.io/projected/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-kube-api-access-h7s4j\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:54.289649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.282190    7082 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:54.289649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.282205    7082 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:54.289649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.282217    7082 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:54.289649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.282231    7082 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:54.289649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.282243    7082 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:10:54.430260 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.429772    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:54.430260 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.429832    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:54.448652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.434238    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"
Oct 29 22:10:54.450021 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.449991    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" event={"ID":"d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b","Type":"ContainerDied","Data":"5c809cdf0ada58600ad854985aa7b6f23ddc3c19de66f7989b770d122d40b832"}
Oct 29 22:10:54.453020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.452997    7082 scope.go:122] "RemoveContainer" containerID="670529ce9d5bcd94f25e790d10960dc5cbd5ebc47e797d59329c8ea6c9bff676"
Oct 29 22:10:54.480699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.480659    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 29 22:10:54.584326 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.584293    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"]
Oct 29 22:10:54.655687 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.655448    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/simulator-w1-a3c02e-kserve-84566984db-4bfmh"]
Oct 29 22:10:54.757885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.757801    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:10:54.757885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.757858    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" podUID="ee700b95-a323-410b-8fc8-1e5198dcf771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:10:54.841651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.840387    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 29 22:10:54.841651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.840447    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 29 22:10:54.953328 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:54.953295    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"
Oct 29 22:10:55.666236 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:55.666203    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" path="/var/lib/kubelet/pods/d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b/volumes"
Oct 29 22:10:55.758244 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:55.758211    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:10:55.758541 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:55.758511    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" podUID="ee700b95-a323-410b-8fc8-1e5198dcf771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:10:55.841696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:55.840037    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 29 22:10:55.841696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:55.840095    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 29 22:10:55.936703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:55.934020    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q" podUID="86f63896-629a-4327-a138-279748c035bd" containerName="payload-pre-processing" containerID="cri-o://89a643d8590a7124072f57fb3793e97014460795b3ed289c8f0172a844ffcfc5" gracePeriod=25
Oct 29 22:10:56.761126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:56.755298    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"]
Oct 29 22:10:56.761126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:56.757575    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 29 22:10:56.761126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:56.757619    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" podUID="ee700b95-a323-410b-8fc8-1e5198dcf771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 29 22:10:56.843667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:56.839337    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 29 22:10:56.843667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:56.839417    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 29 22:10:56.871286 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:56.871248    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x" podUID="79e8c570-3e0f-40b7-9696-81b84b10e8eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:10:57.569679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:57.566795    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:57.569679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:57.566882    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:57.569679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:57.567019    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:10:57.738652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:57.735048    7082 generic.go:356] "Generic (PLEG): container finished" podID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerID="a97556ac71b2b236481c9226e1ba143d21954e41419390bd19a6ca73e9f7a087" exitCode=0
Oct 29 22:10:58.197729 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.194989    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" event={"ID":"c7b934d8-6393-4cfc-882e-07e965ed9d46","Type":"ContainerDied","Data":"a97556ac71b2b236481c9226e1ba143d21954e41419390bd19a6ca73e9f7a087"}
Oct 29 22:10:58.320654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.313971    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:58.320654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.317067    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:58.320654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.317981    7082 scope.go:122] "RemoveContainer" containerID="a97556ac71b2b236481c9226e1ba143d21954e41419390bd19a6ca73e9f7a087"
Oct 29 22:10:58.320654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.319923    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:10:58.322645 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.322578    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:58.322774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.322699    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:58.322774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.322751    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:10:58.447312 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.445714    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:58.448555 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.448486    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:58.452473 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.452116    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:10:58.476876 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:10:58.476842    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:10:58.609411 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.608927    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:58.609411 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.608988    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:58.756059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.755952    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" event={"ID":"c7b934d8-6393-4cfc-882e-07e965ed9d46","Type":"ContainerStarted","Data":"fa8ea1c1e08a261bfc2dddb7a8ff9b454b86a1912b3386a4f42a2ac174cb2c49"}
Oct 29 22:10:58.871426 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:58.871393    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:10:59.386285 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.386193    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"]
Oct 29 22:10:59.387482 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.387459    7082 cpu_manager.go:414] "containerMap: removing container" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main"
Oct 29 22:10:59.387574 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.387486    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main"
Oct 29 22:10:59.387574 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.387530    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main"
Oct 29 22:10:59.387574 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.387538    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main"
Oct 29 22:10:59.387940 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.387904    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ccb6e6-e9bf-4bb0-9a4e-c27b1ae52f5b" containerName="main"
Oct 29 22:10:59.387940 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.387935    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="9900ad31-588e-4077-bef3-1ee57af99f84" containerName="main"
Oct 29 22:10:59.399980 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.399880    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"
Oct 29 22:10:59.408822 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.408766    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"]
Oct 29 22:10:59.421317 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.421289    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:10:59.421757 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.421729    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:10:59.421916 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.421898    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:10:59.422021 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.422005    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:10:59.434450 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.433441    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"
Oct 29 22:10:59.454706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.454676    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"]
Oct 29 22:10:59.506653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.480826    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"]
Oct 29 22:10:59.506653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.495026    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-249a13ac-md4pp\" (UID: \"2250c8f3-cb91-4604-ad64-7c675849dfd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"
Oct 29 22:10:59.506653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.495078    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d687c0-mjvbk\" (UID: \"f0d75350-9a77-4599-bb3f-9d833f84510e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"
Oct 29 22:10:59.506653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.495113    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52gnw\" (UniqueName: \"kubernetes.io/projected/2250c8f3-cb91-4604-ad64-7c675849dfd9-kube-api-access-52gnw\") pod \"maas-api-revoke-sub-249a13ac-md4pp\" (UID: \"2250c8f3-cb91-4604-ad64-7c675849dfd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"
Oct 29 22:10:59.506653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.495391    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6chh\" (UniqueName: \"kubernetes.io/projected/f0d75350-9a77-4599-bb3f-9d833f84510e-kube-api-access-m6chh\") pod \"maas-api-revoke-sub-36d687c0-mjvbk\" (UID: \"f0d75350-9a77-4599-bb3f-9d833f84510e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"
Oct 29 22:10:59.506653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.495462    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-249a13ac-md4pp\" (UID: \"2250c8f3-cb91-4604-ad64-7c675849dfd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"
Oct 29 22:10:59.506653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.495554    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d687c0-mjvbk\" (UID: \"f0d75350-9a77-4599-bb3f-9d833f84510e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"
Oct 29 22:10:59.605651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.599079    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6chh\" (UniqueName: \"kubernetes.io/projected/f0d75350-9a77-4599-bb3f-9d833f84510e-kube-api-access-m6chh\") pod \"maas-api-revoke-sub-36d687c0-mjvbk\" (UID: \"f0d75350-9a77-4599-bb3f-9d833f84510e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"
Oct 29 22:10:59.605651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.599176    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-249a13ac-md4pp\" (UID: \"2250c8f3-cb91-4604-ad64-7c675849dfd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"
Oct 29 22:10:59.605651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.599270    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d687c0-mjvbk\" (UID: \"f0d75350-9a77-4599-bb3f-9d833f84510e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"
Oct 29 22:10:59.605651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.599404    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-249a13ac-md4pp\" (UID: \"2250c8f3-cb91-4604-ad64-7c675849dfd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"
Oct 29 22:10:59.605651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.599439    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d687c0-mjvbk\" (UID: \"f0d75350-9a77-4599-bb3f-9d833f84510e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"
Oct 29 22:10:59.605651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.599466    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52gnw\" (UniqueName: \"kubernetes.io/projected/2250c8f3-cb91-4604-ad64-7c675849dfd9-kube-api-access-52gnw\") pod \"maas-api-revoke-sub-249a13ac-md4pp\" (UID: \"2250c8f3-cb91-4604-ad64-7c675849dfd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"
Oct 29 22:10:59.605651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.600791    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-249a13ac-md4pp\" (UID: \"2250c8f3-cb91-4604-ad64-7c675849dfd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"
Oct 29 22:10:59.623652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.613274    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36d687c0-mjvbk\" (UID: \"f0d75350-9a77-4599-bb3f-9d833f84510e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"
Oct 29 22:10:59.628743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.628543    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52gnw\" (UniqueName: \"kubernetes.io/projected/2250c8f3-cb91-4604-ad64-7c675849dfd9-kube-api-access-52gnw\") pod \"maas-api-revoke-sub-249a13ac-md4pp\" (UID: \"2250c8f3-cb91-4604-ad64-7c675849dfd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"
Oct 29 22:10:59.631985 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.630228    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-249a13ac-md4pp\" (UID: \"2250c8f3-cb91-4604-ad64-7c675849dfd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"
Oct 29 22:10:59.640725 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.638531    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36d687c0-mjvbk\" (UID: \"f0d75350-9a77-4599-bb3f-9d833f84510e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"
Oct 29 22:10:59.641391 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.641363    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6chh\" (UniqueName: \"kubernetes.io/projected/f0d75350-9a77-4599-bb3f-9d833f84510e-kube-api-access-m6chh\") pod \"maas-api-revoke-sub-36d687c0-mjvbk\" (UID: \"f0d75350-9a77-4599-bb3f-9d833f84510e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"
Oct 29 22:10:59.742620 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.742578    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"
Oct 29 22:10:59.786654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:10:59.781590    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"
Oct 29 22:11:00.077874 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:00.077757    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:11:00.077874 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:00.077830    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 29 22:11:00.104650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:00.089564    7082 generic.go:356] "Generic (PLEG): container finished" podID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerID="e3229d11c4b9a6fb6a918a8920bf4458b74679563245fb37b49261b3164cde8f" exitCode=0
Oct 29 22:11:00.131177 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:00.103368    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" event={"ID":"2a7fa2cd-1234-4b80-b869-3ab7019d1cea","Type":"ContainerDied","Data":"e3229d11c4b9a6fb6a918a8920bf4458b74679563245fb37b49261b3164cde8f"}
Oct 29 22:11:00.131177 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:00.128094    7082 scope.go:122] "RemoveContainer" containerID="e3229d11c4b9a6fb6a918a8920bf4458b74679563245fb37b49261b3164cde8f"
Oct 29 22:11:00.133524 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:00.133465    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:11:00.133798 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:00.133764    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:11:00.134119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:00.134086    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:11:00.540139 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:11:00.540103    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2250c8f3_cb91_4604_ad64_7c675849dfd9.slice/crio-01b5cb43cf47c9a111f9c2889e30dd431314e23e68c210bbd56ded9b90243842 WatchSource:0}: Error finding container 01b5cb43cf47c9a111f9c2889e30dd431314e23e68c210bbd56ded9b90243842: Status 404 returned error can't find the container with id 01b5cb43cf47c9a111f9c2889e30dd431314e23e68c210bbd56ded9b90243842
Oct 29 22:11:00.542663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:00.542637    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"]
Oct 29 22:11:00.681126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:00.681091    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"]
Oct 29 22:11:00.757067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:00.757036    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q" podUID="86f63896-629a-4327-a138-279748c035bd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:11:00.871905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:00.871874    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:11:00.872079 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:00.871928    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 29 22:11:01.163751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:01.163671    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk" event={"ID":"f0d75350-9a77-4599-bb3f-9d833f84510e","Type":"ContainerStarted","Data":"0863c5c75c2295465c4b486141d466b416ed3ee42d150a253bc663dff7ddece6"}
Oct 29 22:11:01.166077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:01.166051    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" event={"ID":"2a7fa2cd-1234-4b80-b869-3ab7019d1cea","Type":"ContainerStarted","Data":"75d6369bc74dbba4d07d9a55011e9af9f95fd2df944899d1b2c8908597ad7eec"}
Oct 29 22:11:01.168339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:01.168316    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp" event={"ID":"2250c8f3-cb91-4604-ad64-7c675849dfd9","Type":"ContainerStarted","Data":"e4976ce9a8dddb2824354d814de6643a8a2b7e042eef84935ae100ebcae96168"}
Oct 29 22:11:01.168464 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:01.168358    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp" event={"ID":"2250c8f3-cb91-4604-ad64-7c675849dfd9","Type":"ContainerStarted","Data":"01b5cb43cf47c9a111f9c2889e30dd431314e23e68c210bbd56ded9b90243842"}
Oct 29 22:11:01.258178 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:01.258124    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp" podStartSLOduration=2.258108924 podStartE2EDuration="2.258108924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:01.251877081 +0000 UTC m=+1433.751919380" watchObservedRunningTime="2026-10-29 22:11:01.258108924 +0000 UTC m=+1433.758151223"
Oct 29 22:11:01.903893 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:01.903844    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng" podUID="f65ddf70-7369-430f-bb7d-ac3e2786f92d" containerName="payload-processing" containerID="cri-o://44c48537cfff6ac9d6d0bc036f399f5ec9242599ef22e11acb3c34814b9c0b54" gracePeriod=25
Oct 29 22:11:02.069825 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:01.905611    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:11:02.069825 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.055919    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 29 22:11:02.069825 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:01.956778    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb" podUID="7b205272-e4c0-432c-860d-162d9e80f950" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:11:02.087881 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.082124    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:11:02.087881 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.083175    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:11:02.087881 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.083234    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:11:02.183419 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.183382    7082 generic.go:356] "Generic (PLEG): container finished" podID="2250c8f3-cb91-4604-ad64-7c675849dfd9" containerID="e4976ce9a8dddb2824354d814de6643a8a2b7e042eef84935ae100ebcae96168" exitCode=0
Oct 29 22:11:02.183755 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.183550    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp" event={"ID":"2250c8f3-cb91-4604-ad64-7c675849dfd9","Type":"ContainerDied","Data":"e4976ce9a8dddb2824354d814de6643a8a2b7e042eef84935ae100ebcae96168"}
Oct 29 22:11:02.261015 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.260901    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk" event={"ID":"f0d75350-9a77-4599-bb3f-9d833f84510e","Type":"ContainerStarted","Data":"1852ba3561573d24f4e0fe909d430ab0391d634976d761dac89c590011e86eca"}
Oct 29 22:11:02.353071 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.352987    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk" podStartSLOduration=3.352962409 podStartE2EDuration="3.352962409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:02.33506145 +0000 UTC m=+1434.835103749" watchObservedRunningTime="2026-10-29 22:11:02.352962409 +0000 UTC m=+1434.853004709"
Oct 29 22:11:02.778677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.778597    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"
Oct 29 22:11:02.871204 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.871170    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:11:02.871377 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.871276    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 29 22:11:02.873758 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.873722    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f65ddf70-7369-430f-bb7d-ac3e2786f92d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65ddf70-7369-430f-bb7d-ac3e2786f92d-plugins-config-volume\") pod \"f65ddf70-7369-430f-bb7d-ac3e2786f92d\" (UID: \"f65ddf70-7369-430f-bb7d-ac3e2786f92d\") "
Oct 29 22:11:02.874066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.874049    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f65ddf70-7369-430f-bb7d-ac3e2786f92d-kube-api-access-968sn\" (UniqueName: \"kubernetes.io/projected/f65ddf70-7369-430f-bb7d-ac3e2786f92d-kube-api-access-968sn\") pod \"f65ddf70-7369-430f-bb7d-ac3e2786f92d\" (UID: \"f65ddf70-7369-430f-bb7d-ac3e2786f92d\") "
Oct 29 22:11:02.875349 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.875319    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65ddf70-7369-430f-bb7d-ac3e2786f92d-plugins-config-volume" pod "f65ddf70-7369-430f-bb7d-ac3e2786f92d" (UID: "f65ddf70-7369-430f-bb7d-ac3e2786f92d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:02.876981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.876958    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65ddf70-7369-430f-bb7d-ac3e2786f92d-kube-api-access-968sn" pod "f65ddf70-7369-430f-bb7d-ac3e2786f92d" (UID: "f65ddf70-7369-430f-bb7d-ac3e2786f92d"). InnerVolumeSpecName "kube-api-access-968sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:02.976547 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.976515    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-968sn\" (UniqueName: \"kubernetes.io/projected/f65ddf70-7369-430f-bb7d-ac3e2786f92d-kube-api-access-968sn\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:02.976547 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:02.976542    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65ddf70-7369-430f-bb7d-ac3e2786f92d-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:03.078280 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.078244    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:11:03.078454 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.078307    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:11:03.277156 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.277121    7082 generic.go:356] "Generic (PLEG): container finished" podID="f0d75350-9a77-4599-bb3f-9d833f84510e" containerID="1852ba3561573d24f4e0fe909d430ab0391d634976d761dac89c590011e86eca" exitCode=0
Oct 29 22:11:03.277377 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.277202    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk" event={"ID":"f0d75350-9a77-4599-bb3f-9d833f84510e","Type":"ContainerDied","Data":"1852ba3561573d24f4e0fe909d430ab0391d634976d761dac89c590011e86eca"}
Oct 29 22:11:03.282645 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.281553    7082 generic.go:356] "Generic (PLEG): container finished" podID="f65ddf70-7369-430f-bb7d-ac3e2786f92d" containerID="44c48537cfff6ac9d6d0bc036f399f5ec9242599ef22e11acb3c34814b9c0b54" exitCode=0
Oct 29 22:11:03.282645 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.281871    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"
Oct 29 22:11:03.284147 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.284125    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng" event={"ID":"f65ddf70-7369-430f-bb7d-ac3e2786f92d","Type":"ContainerDied","Data":"44c48537cfff6ac9d6d0bc036f399f5ec9242599ef22e11acb3c34814b9c0b54"}
Oct 29 22:11:03.284270 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.284166    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng" event={"ID":"f65ddf70-7369-430f-bb7d-ac3e2786f92d","Type":"ContainerDied","Data":"12548a1c86aec217181825163c30c0567d5feca5a175312e4f98a835bed92e94"}
Oct 29 22:11:03.284270 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.284189    7082 scope.go:122] "RemoveContainer" containerID="44c48537cfff6ac9d6d0bc036f399f5ec9242599ef22e11acb3c34814b9c0b54"
Oct 29 22:11:03.383244 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.383203    7082 scope.go:122] "RemoveContainer" containerID="44c48537cfff6ac9d6d0bc036f399f5ec9242599ef22e11acb3c34814b9c0b54"
Oct 29 22:11:03.383703 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:03.383562    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c48537cfff6ac9d6d0bc036f399f5ec9242599ef22e11acb3c34814b9c0b54\": container with ID starting with 44c48537cfff6ac9d6d0bc036f399f5ec9242599ef22e11acb3c34814b9c0b54 not found: ID does not exist" containerID="44c48537cfff6ac9d6d0bc036f399f5ec9242599ef22e11acb3c34814b9c0b54"
Oct 29 22:11:03.383703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.383605    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c48537cfff6ac9d6d0bc036f399f5ec9242599ef22e11acb3c34814b9c0b54"} err="failed to get container status \"44c48537cfff6ac9d6d0bc036f399f5ec9242599ef22e11acb3c34814b9c0b54\": rpc error: code = NotFound desc = could not find container \"44c48537cfff6ac9d6d0bc036f399f5ec9242599ef22e11acb3c34814b9c0b54\": container with ID starting with 44c48537cfff6ac9d6d0bc036f399f5ec9242599ef22e11acb3c34814b9c0b54 not found: ID does not exist"
Oct 29 22:11:03.457677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.457611    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"]
Oct 29 22:11:03.465903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.465872    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-86b5d744db-8gkng"]
Oct 29 22:11:03.980216 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.980186    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:11:03.980790 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:03.980761    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 29 22:11:04.078587 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.078542    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:11:04.078861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.078675    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:11:04.082569 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.082536    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f65ddf70-7369-430f-bb7d-ac3e2786f92d" path="/var/lib/kubelet/pods/f65ddf70-7369-430f-bb7d-ac3e2786f92d/volumes"
Oct 29 22:11:04.118653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.107352    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"
Oct 29 22:11:04.162889 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:04.162851    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65ddf70_7369_430f_bb7d_ac3e2786f92d.slice/crio-12548a1c86aec217181825163c30c0567d5feca5a175312e4f98a835bed92e94\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65ddf70_7369_430f_bb7d_ac3e2786f92d.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:04.256727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.255983    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2250c8f3-cb91-4604-ad64-7c675849dfd9-kube-api-access-52gnw\" (UniqueName: \"kubernetes.io/projected/2250c8f3-cb91-4604-ad64-7c675849dfd9-kube-api-access-52gnw\") pod \"2250c8f3-cb91-4604-ad64-7c675849dfd9\" (UID: \"2250c8f3-cb91-4604-ad64-7c675849dfd9\") "
Oct 29 22:11:04.256727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.256132    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-xks-ca\") pod \"2250c8f3-cb91-4604-ad64-7c675849dfd9\" (UID: \"2250c8f3-cb91-4604-ad64-7c675849dfd9\") "
Oct 29 22:11:04.256727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.256518    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-openshift-ca\") pod \"2250c8f3-cb91-4604-ad64-7c675849dfd9\" (UID: \"2250c8f3-cb91-4604-ad64-7c675849dfd9\") "
Oct 29 22:11:04.259015 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.258974    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-openshift-ca" pod "2250c8f3-cb91-4604-ad64-7c675849dfd9" (UID: "2250c8f3-cb91-4604-ad64-7c675849dfd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:04.272912 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.262930    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-xks-ca" pod "2250c8f3-cb91-4604-ad64-7c675849dfd9" (UID: "2250c8f3-cb91-4604-ad64-7c675849dfd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:11:04.273516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.273487    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2250c8f3-cb91-4604-ad64-7c675849dfd9-kube-api-access-52gnw" pod "2250c8f3-cb91-4604-ad64-7c675849dfd9" (UID: "2250c8f3-cb91-4604-ad64-7c675849dfd9"). InnerVolumeSpecName "kube-api-access-52gnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:04.359796 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.359754    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52gnw\" (UniqueName: \"kubernetes.io/projected/2250c8f3-cb91-4604-ad64-7c675849dfd9-kube-api-access-52gnw\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:04.359796 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.359789    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:04.359796 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.359802    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2250c8f3-cb91-4604-ad64-7c675849dfd9-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:04.364393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.364370    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"
Oct 29 22:11:04.364506 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.364401    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp" event={"ID":"2250c8f3-cb91-4604-ad64-7c675849dfd9","Type":"ContainerDied","Data":"01b5cb43cf47c9a111f9c2889e30dd431314e23e68c210bbd56ded9b90243842"}
Oct 29 22:11:04.364506 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.364500    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01b5cb43cf47c9a111f9c2889e30dd431314e23e68c210bbd56ded9b90243842"
Oct 29 22:11:04.761700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.761614    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"
Oct 29 22:11:04.868773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.868736    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-xks-ca\") pod \"f0d75350-9a77-4599-bb3f-9d833f84510e\" (UID: \"f0d75350-9a77-4599-bb3f-9d833f84510e\") "
Oct 29 22:11:04.869068 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.869042    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-openshift-ca\") pod \"f0d75350-9a77-4599-bb3f-9d833f84510e\" (UID: \"f0d75350-9a77-4599-bb3f-9d833f84510e\") "
Oct 29 22:11:04.869168 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.869086    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d75350-9a77-4599-bb3f-9d833f84510e-kube-api-access-m6chh\" (UniqueName: \"kubernetes.io/projected/f0d75350-9a77-4599-bb3f-9d833f84510e-kube-api-access-m6chh\") pod \"f0d75350-9a77-4599-bb3f-9d833f84510e\" (UID: \"f0d75350-9a77-4599-bb3f-9d833f84510e\") "
Oct 29 22:11:04.870282 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.870254    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-openshift-ca" pod "f0d75350-9a77-4599-bb3f-9d833f84510e" (UID: "f0d75350-9a77-4599-bb3f-9d833f84510e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:04.870828 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.870805    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:11:04.870920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.870850    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 29 22:11:04.871422 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.871391    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-xks-ca" pod "f0d75350-9a77-4599-bb3f-9d833f84510e" (UID: "f0d75350-9a77-4599-bb3f-9d833f84510e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:11:04.872522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.872503    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d75350-9a77-4599-bb3f-9d833f84510e-kube-api-access-m6chh" pod "f0d75350-9a77-4599-bb3f-9d833f84510e" (UID: "f0d75350-9a77-4599-bb3f-9d833f84510e"). InnerVolumeSpecName "kube-api-access-m6chh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:04.971396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.971322    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:04.971396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.971350    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d75350-9a77-4599-bb3f-9d833f84510e-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:04.971396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:04.971360    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6chh\" (UniqueName: \"kubernetes.io/projected/f0d75350-9a77-4599-bb3f-9d833f84510e-kube-api-access-m6chh\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:05.078653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:05.078607    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:11:05.079165 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:05.078699    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:11:05.374883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:05.374855    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"
Oct 29 22:11:05.375093 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:05.375069    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk" event={"ID":"f0d75350-9a77-4599-bb3f-9d833f84510e","Type":"ContainerDied","Data":"0863c5c75c2295465c4b486141d466b416ed3ee42d150a253bc663dff7ddece6"}
Oct 29 22:11:05.375192 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:05.375183    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0863c5c75c2295465c4b486141d466b416ed3ee42d150a253bc663dff7ddece6"
Oct 29 22:11:05.892296 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:05.880433    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:11:05.892296 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:05.880492    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 29 22:11:06.079487 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.079452    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:11:06.080113 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.080054    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:11:06.631385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.631350    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"]
Oct 29 22:11:06.632196 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.632174    7082 cpu_manager.go:414] "containerMap: removing container" podUID="f0d75350-9a77-4599-bb3f-9d833f84510e" containerName="revoke-keys"
Oct 29 22:11:06.632196 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.632191    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d75350-9a77-4599-bb3f-9d833f84510e" containerName="revoke-keys"
Oct 29 22:11:06.632402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.632228    7082 cpu_manager.go:414] "containerMap: removing container" podUID="2250c8f3-cb91-4604-ad64-7c675849dfd9" containerName="revoke-keys"
Oct 29 22:11:06.632402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.632233    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2250c8f3-cb91-4604-ad64-7c675849dfd9" containerName="revoke-keys"
Oct 29 22:11:06.632402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.632257    7082 cpu_manager.go:414] "containerMap: removing container" podUID="f65ddf70-7369-430f-bb7d-ac3e2786f92d" containerName="payload-processing"
Oct 29 22:11:06.632402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.632263    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f65ddf70-7369-430f-bb7d-ac3e2786f92d" containerName="payload-processing"
Oct 29 22:11:06.632402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.632374    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="2250c8f3-cb91-4604-ad64-7c675849dfd9" containerName="revoke-keys"
Oct 29 22:11:06.632402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.632388    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="f65ddf70-7369-430f-bb7d-ac3e2786f92d" containerName="payload-processing"
Oct 29 22:11:06.632402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.632397    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d75350-9a77-4599-bb3f-9d833f84510e" containerName="revoke-keys"
Oct 29 22:11:06.636258 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.636236    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"
Oct 29 22:11:06.656470 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.656443    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"]
Oct 29 22:11:06.661204 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.661144    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:11:06.661204 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.661144    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:11:06.661204 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.661176    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:11:06.661804 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.661781    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:11:06.731288 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.731244    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ae129-5czcn\" (UID: \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"
Oct 29 22:11:06.731492 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.731466    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ae129-5czcn\" (UID: \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"
Oct 29 22:11:06.731569 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.731527    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fslzb\" (UniqueName: \"kubernetes.io/projected/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-kube-api-access-fslzb\") pod \"maas-api-revoke-keys-e2e-api-a-9ae129-5czcn\" (UID: \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"
Oct 29 22:11:06.835070 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.833235    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ae129-5czcn\" (UID: \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"
Oct 29 22:11:06.835070 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.833493    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ae129-5czcn\" (UID: \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"
Oct 29 22:11:06.835070 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.833547    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fslzb\" (UniqueName: \"kubernetes.io/projected/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-kube-api-access-fslzb\") pod \"maas-api-revoke-keys-e2e-api-a-9ae129-5czcn\" (UID: \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"
Oct 29 22:11:06.835070 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.834724    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ae129-5czcn\" (UID: \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"
Oct 29 22:11:06.843600 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.843392    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ae129-5czcn\" (UID: \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"
Oct 29 22:11:06.863897 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.863869    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fslzb\" (UniqueName: \"kubernetes.io/projected/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-kube-api-access-fslzb\") pod \"maas-api-revoke-keys-e2e-api-a-9ae129-5czcn\" (UID: \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"
Oct 29 22:11:06.877541 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.877518    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:11:06.878277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.878255    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:11:06.879726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.879708    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:11:06.953170 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.953147    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"
Oct 29 22:11:06.978923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:06.978900    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"
Oct 29 22:11:07.040577 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.040543    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e8c570-3e0f-40b7-9696-81b84b10e8eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e8c570-3e0f-40b7-9696-81b84b10e8eb-plugins-config-volume\") pod \"79e8c570-3e0f-40b7-9696-81b84b10e8eb\" (UID: \"79e8c570-3e0f-40b7-9696-81b84b10e8eb\") "
Oct 29 22:11:07.041163 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.041139    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e8c570-3e0f-40b7-9696-81b84b10e8eb-plugins-config-volume" pod "79e8c570-3e0f-40b7-9696-81b84b10e8eb" (UID: "79e8c570-3e0f-40b7-9696-81b84b10e8eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:07.041473 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.041445    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e8c570-3e0f-40b7-9696-81b84b10e8eb-kube-api-access-vvldb\" (UniqueName: \"kubernetes.io/projected/79e8c570-3e0f-40b7-9696-81b84b10e8eb-kube-api-access-vvldb\") pod \"79e8c570-3e0f-40b7-9696-81b84b10e8eb\" (UID: \"79e8c570-3e0f-40b7-9696-81b84b10e8eb\") "
Oct 29 22:11:07.042847 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.042822    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e8c570-3e0f-40b7-9696-81b84b10e8eb-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:07.046392 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.046261    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e8c570-3e0f-40b7-9696-81b84b10e8eb-kube-api-access-vvldb" pod "79e8c570-3e0f-40b7-9696-81b84b10e8eb" (UID: "79e8c570-3e0f-40b7-9696-81b84b10e8eb"). InnerVolumeSpecName "kube-api-access-vvldb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:07.078487 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.078406    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:11:07.078487 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.078471    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:11:07.167673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.158702    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvldb\" (UniqueName: \"kubernetes.io/projected/79e8c570-3e0f-40b7-9696-81b84b10e8eb-kube-api-access-vvldb\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:07.362794 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.319521    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"]
Oct 29 22:11:07.656993 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.656212    7082 cpu_manager.go:414] "containerMap: removing container" podUID="79e8c570-3e0f-40b7-9696-81b84b10e8eb" containerName="payload-processing"
Oct 29 22:11:07.656993 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.656241    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e8c570-3e0f-40b7-9696-81b84b10e8eb" containerName="payload-processing"
Oct 29 22:11:07.659698 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.657535    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e8c570-3e0f-40b7-9696-81b84b10e8eb" containerName="payload-processing"
Oct 29 22:11:07.744764 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.744730    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"
Oct 29 22:11:07.758698 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.758550    7082 generic.go:356] "Generic (PLEG): container finished" podID="79e8c570-3e0f-40b7-9696-81b84b10e8eb" containerID="e3ab74d6d6c07120f1861fbc0a0b1cea3fe8f7e62a72f6f6ded5d2e329656226" exitCode=0
Oct 29 22:11:07.763168 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.758938    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"
Oct 29 22:11:07.764036 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.763997    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x" event={"ID":"79e8c570-3e0f-40b7-9696-81b84b10e8eb","Type":"ContainerDied","Data":"e3ab74d6d6c07120f1861fbc0a0b1cea3fe8f7e62a72f6f6ded5d2e329656226"}
Oct 29 22:11:07.764145 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.764067    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x" event={"ID":"79e8c570-3e0f-40b7-9696-81b84b10e8eb","Type":"ContainerDied","Data":"a0319ca7d34e60afd0961c975fed38c6a178be012e98c790d5805f2d9f3fc154"}
Oct 29 22:11:07.764219 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.764180    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"]
Oct 29 22:11:07.765085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.764888    7082 scope.go:122] "RemoveContainer" containerID="e3ab74d6d6c07120f1861fbc0a0b1cea3fe8f7e62a72f6f6ded5d2e329656226"
Oct 29 22:11:07.771567 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.771548    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"]
Oct 29 22:11:07.771732 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.771590    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"]
Oct 29 22:11:07.771732 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.771681    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"
Oct 29 22:11:07.788865 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.788836    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bac048a8-mkkrm\" (UID: \"b319e77b-08ea-4820-99f9-98859acbbcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"
Oct 29 22:11:07.788943 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.788925    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75f87b5d-hdvc4\" (UID: \"cd3b72f8-23bb-4617-a074-384c32e35667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"
Oct 29 22:11:07.788980 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.788955    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f84qc\" (UniqueName: \"kubernetes.io/projected/cd3b72f8-23bb-4617-a074-384c32e35667-kube-api-access-f84qc\") pod \"maas-api-revoke-sub-75f87b5d-hdvc4\" (UID: \"cd3b72f8-23bb-4617-a074-384c32e35667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"
Oct 29 22:11:07.789178 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.789158    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75f87b5d-hdvc4\" (UID: \"cd3b72f8-23bb-4617-a074-384c32e35667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"
Oct 29 22:11:07.789275 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.789259    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bac048a8-mkkrm\" (UID: \"b319e77b-08ea-4820-99f9-98859acbbcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"
Oct 29 22:11:07.789563 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.789544    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhcqh\" (UniqueName: \"kubernetes.io/projected/b319e77b-08ea-4820-99f9-98859acbbcbb-kube-api-access-rhcqh\") pod \"maas-api-revoke-sub-bac048a8-mkkrm\" (UID: \"b319e77b-08ea-4820-99f9-98859acbbcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"
Oct 29 22:11:07.793122 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.793073    7082 scope.go:122] "RemoveContainer" containerID="e3ab74d6d6c07120f1861fbc0a0b1cea3fe8f7e62a72f6f6ded5d2e329656226"
Oct 29 22:11:07.793760 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:07.793730    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ab74d6d6c07120f1861fbc0a0b1cea3fe8f7e62a72f6f6ded5d2e329656226\": container with ID starting with e3ab74d6d6c07120f1861fbc0a0b1cea3fe8f7e62a72f6f6ded5d2e329656226 not found: ID does not exist" containerID="e3ab74d6d6c07120f1861fbc0a0b1cea3fe8f7e62a72f6f6ded5d2e329656226"
Oct 29 22:11:07.793844 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.793807    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ab74d6d6c07120f1861fbc0a0b1cea3fe8f7e62a72f6f6ded5d2e329656226"} err="failed to get container status \"e3ab74d6d6c07120f1861fbc0a0b1cea3fe8f7e62a72f6f6ded5d2e329656226\": rpc error: code = NotFound desc = could not find container \"e3ab74d6d6c07120f1861fbc0a0b1cea3fe8f7e62a72f6f6ded5d2e329656226\": container with ID starting with e3ab74d6d6c07120f1861fbc0a0b1cea3fe8f7e62a72f6f6ded5d2e329656226 not found: ID does not exist"
Oct 29 22:11:07.884718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.876822    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x" podUID="79e8c570-3e0f-40b7-9696-81b84b10e8eb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.72:50052\" within 1s: context deadline exceeded"
Oct 29 22:11:07.884718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.876932    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"
Oct 29 22:11:07.884718 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:11:07.877049    7082 logging.go:55] [core] [Channel #2289 SubChannel #2290]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.72:50052", ServerName: "10.134.0.72:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.72:50052: operation was canceled"
Oct 29 22:11:07.911653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.898215    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bac048a8-mkkrm\" (UID: \"b319e77b-08ea-4820-99f9-98859acbbcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"
Oct 29 22:11:07.911653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.898342    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75f87b5d-hdvc4\" (UID: \"cd3b72f8-23bb-4617-a074-384c32e35667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"
Oct 29 22:11:07.911653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.898375    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f84qc\" (UniqueName: \"kubernetes.io/projected/cd3b72f8-23bb-4617-a074-384c32e35667-kube-api-access-f84qc\") pod \"maas-api-revoke-sub-75f87b5d-hdvc4\" (UID: \"cd3b72f8-23bb-4617-a074-384c32e35667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"
Oct 29 22:11:07.911653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.898526    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75f87b5d-hdvc4\" (UID: \"cd3b72f8-23bb-4617-a074-384c32e35667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"
Oct 29 22:11:07.911653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.898597    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bac048a8-mkkrm\" (UID: \"b319e77b-08ea-4820-99f9-98859acbbcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"
Oct 29 22:11:07.911653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.898827    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhcqh\" (UniqueName: \"kubernetes.io/projected/b319e77b-08ea-4820-99f9-98859acbbcbb-kube-api-access-rhcqh\") pod \"maas-api-revoke-sub-bac048a8-mkkrm\" (UID: \"b319e77b-08ea-4820-99f9-98859acbbcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"
Oct 29 22:11:07.911653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.905064    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75f87b5d-hdvc4\" (UID: \"cd3b72f8-23bb-4617-a074-384c32e35667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"
Oct 29 22:11:07.929522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.929486    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bac048a8-mkkrm\" (UID: \"b319e77b-08ea-4820-99f9-98859acbbcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"
Oct 29 22:11:07.976426 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.975991    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f84qc\" (UniqueName: \"kubernetes.io/projected/cd3b72f8-23bb-4617-a074-384c32e35667-kube-api-access-f84qc\") pod \"maas-api-revoke-sub-75f87b5d-hdvc4\" (UID: \"cd3b72f8-23bb-4617-a074-384c32e35667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"
Oct 29 22:11:07.976426 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.976376    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75f87b5d-hdvc4\" (UID: \"cd3b72f8-23bb-4617-a074-384c32e35667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"
Oct 29 22:11:07.988974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.981463    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhcqh\" (UniqueName: \"kubernetes.io/projected/b319e77b-08ea-4820-99f9-98859acbbcbb-kube-api-access-rhcqh\") pod \"maas-api-revoke-sub-bac048a8-mkkrm\" (UID: \"b319e77b-08ea-4820-99f9-98859acbbcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"
Oct 29 22:11:07.988974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:07.986190    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bac048a8-mkkrm\" (UID: \"b319e77b-08ea-4820-99f9-98859acbbcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"
Oct 29 22:11:08.020966 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:11:08.020534    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9676fbf_6ef6_4ad1_a04c_5a962df40ac8.slice/crio-e8db5e22ebf06eeb315ea25f3a8e7b378b93c7eae3296a4dc8c7234b623611e8 WatchSource:0}: Error finding container e8db5e22ebf06eeb315ea25f3a8e7b378b93c7eae3296a4dc8c7234b623611e8: Status 404 returned error can't find the container with id e8db5e22ebf06eeb315ea25f3a8e7b378b93c7eae3296a4dc8c7234b623611e8
Oct 29 22:11:08.034027 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.033997    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"]
Oct 29 22:11:08.043707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.043663    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a9def7-7dd45d5986-vb64x"]
Oct 29 22:11:08.051583 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.051550    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"]
Oct 29 22:11:08.077067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.077027    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"
Oct 29 22:11:08.078101 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.078076    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Oct 29 22:11:08.078574 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.078126    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Oct 29 22:11:08.105111 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.105077    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"
Oct 29 22:11:08.501999 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.501973    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"]
Oct 29 22:11:08.502776 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:11:08.502172    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb319e77b_08ea_4820_99f9_98859acbbcbb.slice/crio-7e75b0c9027fe7e9deac4940912a711008ac59209d0a9896144711dd25934194 WatchSource:0}: Error finding container 7e75b0c9027fe7e9deac4940912a711008ac59209d0a9896144711dd25934194: Status 404 returned error can't find the container with id 7e75b0c9027fe7e9deac4940912a711008ac59209d0a9896144711dd25934194
Oct 29 22:11:08.515514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.515491    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"]
Oct 29 22:11:08.522869 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:11:08.522839    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3b72f8_23bb_4617_a074_384c32e35667.slice/crio-18bc7748d0e0af2bbe16dce15b07fcac5f3c9b2e777f417dcf16328c6c1bdac6 WatchSource:0}: Error finding container 18bc7748d0e0af2bbe16dce15b07fcac5f3c9b2e777f417dcf16328c6c1bdac6: Status 404 returned error can't find the container with id 18bc7748d0e0af2bbe16dce15b07fcac5f3c9b2e777f417dcf16328c6c1bdac6
Oct 29 22:11:08.769768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.769679    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn" event={"ID":"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8","Type":"ContainerStarted","Data":"4b39db52d109c7b8bbece46deb9570a5eb17bf67089159f79a7d6f6181eb807c"}
Oct 29 22:11:08.769768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.769728    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn" event={"ID":"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8","Type":"ContainerStarted","Data":"e8db5e22ebf06eeb315ea25f3a8e7b378b93c7eae3296a4dc8c7234b623611e8"}
Oct 29 22:11:08.771147 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.771119    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm" event={"ID":"b319e77b-08ea-4820-99f9-98859acbbcbb","Type":"ContainerStarted","Data":"7e75b0c9027fe7e9deac4940912a711008ac59209d0a9896144711dd25934194"}
Oct 29 22:11:08.772677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.772654    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4" event={"ID":"cd3b72f8-23bb-4617-a074-384c32e35667","Type":"ContainerStarted","Data":"18bc7748d0e0af2bbe16dce15b07fcac5f3c9b2e777f417dcf16328c6c1bdac6"}
Oct 29 22:11:08.813337 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:08.813294    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn" podStartSLOduration=2.813280415 podStartE2EDuration="2.813280415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:08.808944091 +0000 UTC m=+1441.308986390" watchObservedRunningTime="2026-10-29 22:11:08.813280415 +0000 UTC m=+1441.313322717"
Oct 29 22:11:09.083155 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:09.083131    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:11:09.083615 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:09.083601    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:11:09.084395 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:09.084377    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:11:09.708930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:09.704673    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e8c570-3e0f-40b7-9696-81b84b10e8eb" path="/var/lib/kubelet/pods/79e8c570-3e0f-40b7-9696-81b84b10e8eb/volumes"
Oct 29 22:11:09.783923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:09.783844    7082 generic.go:356] "Generic (PLEG): container finished" podID="b319e77b-08ea-4820-99f9-98859acbbcbb" containerID="e8f57a6d149a128a183972fe2f512087d6fe130c7e3f6e1ba3ebd7bf0f74291a" exitCode=0
Oct 29 22:11:09.784143 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:09.783929    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm" event={"ID":"b319e77b-08ea-4820-99f9-98859acbbcbb","Type":"ContainerDied","Data":"e8f57a6d149a128a183972fe2f512087d6fe130c7e3f6e1ba3ebd7bf0f74291a"}
Oct 29 22:11:09.785925 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:09.785904    7082 generic.go:356] "Generic (PLEG): container finished" podID="cd3b72f8-23bb-4617-a074-384c32e35667" containerID="cc4de09ac7b721e4cb6fbbbf71caf0ce8130d9bacd17ba8e2c9f8f3d0933031c" exitCode=0
Oct 29 22:11:09.786018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:09.785986    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4" event={"ID":"cd3b72f8-23bb-4617-a074-384c32e35667","Type":"ContainerDied","Data":"cc4de09ac7b721e4cb6fbbbf71caf0ce8130d9bacd17ba8e2c9f8f3d0933031c"}
Oct 29 22:11:09.787607 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:09.787587    7082 generic.go:356] "Generic (PLEG): container finished" podID="f9676fbf-6ef6-4ad1-a04c-5a962df40ac8" containerID="4b39db52d109c7b8bbece46deb9570a5eb17bf67089159f79a7d6f6181eb807c" exitCode=0
Oct 29 22:11:09.787721 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:09.787675    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn" event={"ID":"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8","Type":"ContainerDied","Data":"4b39db52d109c7b8bbece46deb9570a5eb17bf67089159f79a7d6f6181eb807c"}
Oct 29 22:11:10.794882 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:10.788391    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q" podUID="86f63896-629a-4327-a138-279748c035bd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:11:11.267796 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.266852    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"
Oct 29 22:11:11.337768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.337739    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"
Oct 29 22:11:11.339757 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.339736    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"
Oct 29 22:11:11.396278 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.396194    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-xks-ca\") pod \"cd3b72f8-23bb-4617-a074-384c32e35667\" (UID: \"cd3b72f8-23bb-4617-a074-384c32e35667\") "
Oct 29 22:11:11.396495 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.396445    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd3b72f8-23bb-4617-a074-384c32e35667-kube-api-access-f84qc\" (UniqueName: \"kubernetes.io/projected/cd3b72f8-23bb-4617-a074-384c32e35667-kube-api-access-f84qc\") pod \"cd3b72f8-23bb-4617-a074-384c32e35667\" (UID: \"cd3b72f8-23bb-4617-a074-384c32e35667\") "
Oct 29 22:11:11.396578 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.396550    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-openshift-ca\") pod \"cd3b72f8-23bb-4617-a074-384c32e35667\" (UID: \"cd3b72f8-23bb-4617-a074-384c32e35667\") "
Oct 29 22:11:11.397309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.397280    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-openshift-ca" pod "cd3b72f8-23bb-4617-a074-384c32e35667" (UID: "cd3b72f8-23bb-4617-a074-384c32e35667"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:11.397597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.397576    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:11.409701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.399045    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-xks-ca" pod "cd3b72f8-23bb-4617-a074-384c32e35667" (UID: "cd3b72f8-23bb-4617-a074-384c32e35667"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:11:11.409701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.400010    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3b72f8-23bb-4617-a074-384c32e35667-kube-api-access-f84qc" pod "cd3b72f8-23bb-4617-a074-384c32e35667" (UID: "cd3b72f8-23bb-4617-a074-384c32e35667"). InnerVolumeSpecName "kube-api-access-f84qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:11.531229 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.531190    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-kube-api-access-fslzb\" (UniqueName: \"kubernetes.io/projected/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-kube-api-access-fslzb\") pod \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\" (UID: \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\") "
Oct 29 22:11:11.597486 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.542298    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-kube-api-access-fslzb" pod "f9676fbf-6ef6-4ad1-a04c-5a962df40ac8" (UID: "f9676fbf-6ef6-4ad1-a04c-5a962df40ac8"). InnerVolumeSpecName "kube-api-access-fslzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:11.597776 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.597599    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-xks-ca\") pod \"b319e77b-08ea-4820-99f9-98859acbbcbb\" (UID: \"b319e77b-08ea-4820-99f9-98859acbbcbb\") "
Oct 29 22:11:11.598057 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.598033    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-xks-ca\") pod \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\" (UID: \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\") "
Oct 29 22:11:11.598873 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.598852    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-openshift-ca\") pod \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\" (UID: \"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8\") "
Oct 29 22:11:11.602105 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.599062    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b319e77b-08ea-4820-99f9-98859acbbcbb-kube-api-access-rhcqh\" (UniqueName: \"kubernetes.io/projected/b319e77b-08ea-4820-99f9-98859acbbcbb-kube-api-access-rhcqh\") pod \"b319e77b-08ea-4820-99f9-98859acbbcbb\" (UID: \"b319e77b-08ea-4820-99f9-98859acbbcbb\") "
Oct 29 22:11:11.602446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.602387    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-openshift-ca" pod "f9676fbf-6ef6-4ad1-a04c-5a962df40ac8" (UID: "f9676fbf-6ef6-4ad1-a04c-5a962df40ac8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:11.602708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.602602    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-openshift-ca\") pod \"b319e77b-08ea-4820-99f9-98859acbbcbb\" (UID: \"b319e77b-08ea-4820-99f9-98859acbbcbb\") "
Oct 29 22:11:11.604032 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.603940    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f84qc\" (UniqueName: \"kubernetes.io/projected/cd3b72f8-23bb-4617-a074-384c32e35667-kube-api-access-f84qc\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:11.604032 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.603975    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:11.604032 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.603998    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd3b72f8-23bb-4617-a074-384c32e35667-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:11.604032 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.604012    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fslzb\" (UniqueName: \"kubernetes.io/projected/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-kube-api-access-fslzb\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:11.604611 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.604311    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-openshift-ca" pod "b319e77b-08ea-4820-99f9-98859acbbcbb" (UID: "b319e77b-08ea-4820-99f9-98859acbbcbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:11.608669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.608616    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b319e77b-08ea-4820-99f9-98859acbbcbb-kube-api-access-rhcqh" pod "b319e77b-08ea-4820-99f9-98859acbbcbb" (UID: "b319e77b-08ea-4820-99f9-98859acbbcbb"). InnerVolumeSpecName "kube-api-access-rhcqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:11.609524 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.609495    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-xks-ca" pod "b319e77b-08ea-4820-99f9-98859acbbcbb" (UID: "b319e77b-08ea-4820-99f9-98859acbbcbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:11:11.609695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.609679    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-xks-ca" pod "f9676fbf-6ef6-4ad1-a04c-5a962df40ac8" (UID: "f9676fbf-6ef6-4ad1-a04c-5a962df40ac8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:11:11.706844 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.706814    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhcqh\" (UniqueName: \"kubernetes.io/projected/b319e77b-08ea-4820-99f9-98859acbbcbb-kube-api-access-rhcqh\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:11.706844 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.706847    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:11.707013 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.706863    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b319e77b-08ea-4820-99f9-98859acbbcbb-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:11.707013 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.706876    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:11.758792 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.758750    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb" podUID="7b205272-e4c0-432c-860d-162d9e80f950" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:11:11.811192 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.811170    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"
Oct 29 22:11:11.811621 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.811257    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4" event={"ID":"cd3b72f8-23bb-4617-a074-384c32e35667","Type":"ContainerDied","Data":"18bc7748d0e0af2bbe16dce15b07fcac5f3c9b2e777f417dcf16328c6c1bdac6"}
Oct 29 22:11:11.811621 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.811299    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18bc7748d0e0af2bbe16dce15b07fcac5f3c9b2e777f417dcf16328c6c1bdac6"
Oct 29 22:11:11.812946 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.812923    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"
Oct 29 22:11:11.812946 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.812931    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn" event={"ID":"f9676fbf-6ef6-4ad1-a04c-5a962df40ac8","Type":"ContainerDied","Data":"e8db5e22ebf06eeb315ea25f3a8e7b378b93c7eae3296a4dc8c7234b623611e8"}
Oct 29 22:11:11.813122 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.812964    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8db5e22ebf06eeb315ea25f3a8e7b378b93c7eae3296a4dc8c7234b623611e8"
Oct 29 22:11:11.814604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.814588    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"
Oct 29 22:11:11.814753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.814602    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm" event={"ID":"b319e77b-08ea-4820-99f9-98859acbbcbb","Type":"ContainerDied","Data":"7e75b0c9027fe7e9deac4940912a711008ac59209d0a9896144711dd25934194"}
Oct 29 22:11:11.814753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:11.814664    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e75b0c9027fe7e9deac4940912a711008ac59209d0a9896144711dd25934194"
Oct 29 22:11:14.257690 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:14.257646    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8c570_3e0f_40b7_9696_81b84b10e8eb.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:14.704465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.704435    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"
Oct 29 22:11:14.787609 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.787574    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b205272-e4c0-432c-860d-162d9e80f950-kube-api-access-jxsjb\" (UniqueName: \"kubernetes.io/projected/7b205272-e4c0-432c-860d-162d9e80f950-kube-api-access-jxsjb\") pod \"7b205272-e4c0-432c-860d-162d9e80f950\" (UID: \"7b205272-e4c0-432c-860d-162d9e80f950\") "
Oct 29 22:11:14.787797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.787660    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b205272-e4c0-432c-860d-162d9e80f950-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b205272-e4c0-432c-860d-162d9e80f950-plugins-config-volume\") pod \"7b205272-e4c0-432c-860d-162d9e80f950\" (UID: \"7b205272-e4c0-432c-860d-162d9e80f950\") "
Oct 29 22:11:14.788160 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.788127    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b205272-e4c0-432c-860d-162d9e80f950-plugins-config-volume" pod "7b205272-e4c0-432c-860d-162d9e80f950" (UID: "7b205272-e4c0-432c-860d-162d9e80f950"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:14.788873 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.788849    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b205272-e4c0-432c-860d-162d9e80f950-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:14.790143 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.790120    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b205272-e4c0-432c-860d-162d9e80f950-kube-api-access-jxsjb" pod "7b205272-e4c0-432c-860d-162d9e80f950" (UID: "7b205272-e4c0-432c-860d-162d9e80f950"). InnerVolumeSpecName "kube-api-access-jxsjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:14.890326 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.890252    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxsjb\" (UniqueName: \"kubernetes.io/projected/7b205272-e4c0-432c-860d-162d9e80f950-kube-api-access-jxsjb\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:14.919226 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.919191    7082 generic.go:356] "Generic (PLEG): container finished" podID="7b205272-e4c0-432c-860d-162d9e80f950" containerID="f7cc9d56f53ee9826288e4c70970cd7d3f9f94fec4d1b0011c1aa09bf88f511c" exitCode=0
Oct 29 22:11:14.919382 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.919232    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"
Oct 29 22:11:14.919382 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.919266    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb" event={"ID":"7b205272-e4c0-432c-860d-162d9e80f950","Type":"ContainerDied","Data":"f7cc9d56f53ee9826288e4c70970cd7d3f9f94fec4d1b0011c1aa09bf88f511c"}
Oct 29 22:11:14.919382 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.919318    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb" event={"ID":"7b205272-e4c0-432c-860d-162d9e80f950","Type":"ContainerDied","Data":"f53ee408e0caa4b804d12935b010e527a23ecca64d086b1aa722114b79bec96b"}
Oct 29 22:11:14.919382 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.919346    7082 scope.go:122] "RemoveContainer" containerID="f7cc9d56f53ee9826288e4c70970cd7d3f9f94fec4d1b0011c1aa09bf88f511c"
Oct 29 22:11:14.935119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.935099    7082 scope.go:122] "RemoveContainer" containerID="f7cc9d56f53ee9826288e4c70970cd7d3f9f94fec4d1b0011c1aa09bf88f511c"
Oct 29 22:11:14.935435 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:14.935411    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7cc9d56f53ee9826288e4c70970cd7d3f9f94fec4d1b0011c1aa09bf88f511c\": container with ID starting with f7cc9d56f53ee9826288e4c70970cd7d3f9f94fec4d1b0011c1aa09bf88f511c not found: ID does not exist" containerID="f7cc9d56f53ee9826288e4c70970cd7d3f9f94fec4d1b0011c1aa09bf88f511c"
Oct 29 22:11:14.935510 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.935446    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7cc9d56f53ee9826288e4c70970cd7d3f9f94fec4d1b0011c1aa09bf88f511c"} err="failed to get container status \"f7cc9d56f53ee9826288e4c70970cd7d3f9f94fec4d1b0011c1aa09bf88f511c\": rpc error: code = NotFound desc = could not find container \"f7cc9d56f53ee9826288e4c70970cd7d3f9f94fec4d1b0011c1aa09bf88f511c\": container with ID starting with f7cc9d56f53ee9826288e4c70970cd7d3f9f94fec4d1b0011c1aa09bf88f511c not found: ID does not exist"
Oct 29 22:11:14.975080 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.975054    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"]
Oct 29 22:11:14.979679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:14.979654    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a3c02e-6f49487c76-htrsb"]
Oct 29 22:11:16.147995 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:16.065314    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8c570_3e0f_40b7_9696_81b84b10e8eb.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:16.147995 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.106811    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b205272-e4c0-432c-860d-162d9e80f950" path="/var/lib/kubelet/pods/7b205272-e4c0-432c-860d-162d9e80f950/volumes"
Oct 29 22:11:16.147995 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:16.109787    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8c570_3e0f_40b7_9696_81b84b10e8eb.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:16.258806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.254412    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"]
Oct 29 22:11:16.258806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.256708    7082 cpu_manager.go:414] "containerMap: removing container" podUID="7b205272-e4c0-432c-860d-162d9e80f950" containerName="payload-processing"
Oct 29 22:11:16.258806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.256731    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b205272-e4c0-432c-860d-162d9e80f950" containerName="payload-processing"
Oct 29 22:11:16.258806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.256775    7082 cpu_manager.go:414] "containerMap: removing container" podUID="f9676fbf-6ef6-4ad1-a04c-5a962df40ac8" containerName="revoke-keys"
Oct 29 22:11:16.258806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.256783    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9676fbf-6ef6-4ad1-a04c-5a962df40ac8" containerName="revoke-keys"
Oct 29 22:11:16.258806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.256808    7082 cpu_manager.go:414] "containerMap: removing container" podUID="b319e77b-08ea-4820-99f9-98859acbbcbb" containerName="revoke-keys"
Oct 29 22:11:16.258806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.256815    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b319e77b-08ea-4820-99f9-98859acbbcbb" containerName="revoke-keys"
Oct 29 22:11:16.258806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.256902    7082 cpu_manager.go:414] "containerMap: removing container" podUID="cd3b72f8-23bb-4617-a074-384c32e35667" containerName="revoke-keys"
Oct 29 22:11:16.258806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.256909    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3b72f8-23bb-4617-a074-384c32e35667" containerName="revoke-keys"
Oct 29 22:11:16.258806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.257374    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b205272-e4c0-432c-860d-162d9e80f950" containerName="payload-processing"
Oct 29 22:11:16.258806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.257398    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9676fbf-6ef6-4ad1-a04c-5a962df40ac8" containerName="revoke-keys"
Oct 29 22:11:16.258806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.257409    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="b319e77b-08ea-4820-99f9-98859acbbcbb" containerName="revoke-keys"
Oct 29 22:11:16.258806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.257436    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3b72f8-23bb-4617-a074-384c32e35667" containerName="revoke-keys"
Oct 29 22:11:16.272448 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.272420    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"]
Oct 29 22:11:16.274077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.273794    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"
Oct 29 22:11:16.292818 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.291935    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f82846\""
Oct 29 22:11:16.292818 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.292059    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f82846-dockercfg-gjjwb\""
Oct 29 22:11:16.367313 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.367282    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67c489e-b41a-4414-9045-8a435b773168-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2\" (UID: \"a67c489e-b41a-4414-9045-8a435b773168\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"
Oct 29 22:11:16.367313 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.367326    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pkh8\" (UniqueName: \"kubernetes.io/projected/a67c489e-b41a-4414-9045-8a435b773168-kube-api-access-6pkh8\") pod \"payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2\" (UID: \"a67c489e-b41a-4414-9045-8a435b773168\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"
Oct 29 22:11:16.466291 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.466265    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"
Oct 29 22:11:16.469719 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.469697    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67c489e-b41a-4414-9045-8a435b773168-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2\" (UID: \"a67c489e-b41a-4414-9045-8a435b773168\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"
Oct 29 22:11:16.469825 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.469734    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pkh8\" (UniqueName: \"kubernetes.io/projected/a67c489e-b41a-4414-9045-8a435b773168-kube-api-access-6pkh8\") pod \"payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2\" (UID: \"a67c489e-b41a-4414-9045-8a435b773168\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"
Oct 29 22:11:16.470329 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.470310    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67c489e-b41a-4414-9045-8a435b773168-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2\" (UID: \"a67c489e-b41a-4414-9045-8a435b773168\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"
Oct 29 22:11:16.492609 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.492579    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pkh8\" (UniqueName: \"kubernetes.io/projected/a67c489e-b41a-4414-9045-8a435b773168-kube-api-access-6pkh8\") pod \"payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2\" (UID: \"a67c489e-b41a-4414-9045-8a435b773168\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"
Oct 29 22:11:16.571444 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.571413    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f63896-629a-4327-a138-279748c035bd-kube-api-access-rxdlc\" (UniqueName: \"kubernetes.io/projected/86f63896-629a-4327-a138-279748c035bd-kube-api-access-rxdlc\") pod \"86f63896-629a-4327-a138-279748c035bd\" (UID: \"86f63896-629a-4327-a138-279748c035bd\") "
Oct 29 22:11:16.571668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.571595    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86f63896-629a-4327-a138-279748c035bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f63896-629a-4327-a138-279748c035bd-plugins-config-volume\") pod \"86f63896-629a-4327-a138-279748c035bd\" (UID: \"86f63896-629a-4327-a138-279748c035bd\") "
Oct 29 22:11:16.572056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.572029    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f63896-629a-4327-a138-279748c035bd-plugins-config-volume" pod "86f63896-629a-4327-a138-279748c035bd" (UID: "86f63896-629a-4327-a138-279748c035bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:16.572620 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.572595    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f63896-629a-4327-a138-279748c035bd-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:16.574303 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.574279    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f63896-629a-4327-a138-279748c035bd-kube-api-access-rxdlc" pod "86f63896-629a-4327-a138-279748c035bd" (UID: "86f63896-629a-4327-a138-279748c035bd"). InnerVolumeSpecName "kube-api-access-rxdlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:16.599405 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.599373    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"
Oct 29 22:11:16.689719 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:16.679659    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxdlc\" (UniqueName: \"kubernetes.io/projected/86f63896-629a-4327-a138-279748c035bd-kube-api-access-rxdlc\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:17.081479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.081451    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"]
Oct 29 22:11:17.083249 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:11:17.083221    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67c489e_b41a_4414_9045_8a435b773168.slice/crio-92abdd5896e3369a83fe21f4cb9390538c877bd7e6e04ca73baff9906f210817 WatchSource:0}: Error finding container 92abdd5896e3369a83fe21f4cb9390538c877bd7e6e04ca73baff9906f210817: Status 404 returned error can't find the container with id 92abdd5896e3369a83fe21f4cb9390538c877bd7e6e04ca73baff9906f210817
Oct 29 22:11:17.300709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.300063    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2" event={"ID":"a67c489e-b41a-4414-9045-8a435b773168","Type":"ContainerStarted","Data":"cefd1fad79f38d8dfaf8ffb662a63c457613d3da1c6e2cfaadc56519355e0fc2"}
Oct 29 22:11:17.300709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.300135    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2" event={"ID":"a67c489e-b41a-4414-9045-8a435b773168","Type":"ContainerStarted","Data":"92abdd5896e3369a83fe21f4cb9390538c877bd7e6e04ca73baff9906f210817"}
Oct 29 22:11:17.300709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.300339    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"
Oct 29 22:11:17.303845 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.303816    7082 generic.go:356] "Generic (PLEG): container finished" podID="86f63896-629a-4327-a138-279748c035bd" containerID="89a643d8590a7124072f57fb3793e97014460795b3ed289c8f0172a844ffcfc5" exitCode=0
Oct 29 22:11:17.304224 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.304154    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"
Oct 29 22:11:17.304781 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.304173    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q" event={"ID":"86f63896-629a-4327-a138-279748c035bd","Type":"ContainerDied","Data":"89a643d8590a7124072f57fb3793e97014460795b3ed289c8f0172a844ffcfc5"}
Oct 29 22:11:17.304944 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.304927    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q" event={"ID":"86f63896-629a-4327-a138-279748c035bd","Type":"ContainerDied","Data":"8f25816e7a98a60194f6eab1f6ffe580627a0de32b9df76bffab89ddb76ca85b"}
Oct 29 22:11:17.305087 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.304995    7082 scope.go:122] "RemoveContainer" containerID="89a643d8590a7124072f57fb3793e97014460795b3ed289c8f0172a844ffcfc5"
Oct 29 22:11:17.323344 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.323328    7082 scope.go:122] "RemoveContainer" containerID="89a643d8590a7124072f57fb3793e97014460795b3ed289c8f0172a844ffcfc5"
Oct 29 22:11:17.324163 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:17.324143    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a643d8590a7124072f57fb3793e97014460795b3ed289c8f0172a844ffcfc5\": container with ID starting with 89a643d8590a7124072f57fb3793e97014460795b3ed289c8f0172a844ffcfc5 not found: ID does not exist" containerID="89a643d8590a7124072f57fb3793e97014460795b3ed289c8f0172a844ffcfc5"
Oct 29 22:11:17.324238 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.324171    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a643d8590a7124072f57fb3793e97014460795b3ed289c8f0172a844ffcfc5"} err="failed to get container status \"89a643d8590a7124072f57fb3793e97014460795b3ed289c8f0172a844ffcfc5\": rpc error: code = NotFound desc = could not find container \"89a643d8590a7124072f57fb3793e97014460795b3ed289c8f0172a844ffcfc5\": container with ID starting with 89a643d8590a7124072f57fb3793e97014460795b3ed289c8f0172a844ffcfc5 not found: ID does not exist"
Oct 29 22:11:17.348823 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.348727    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2" podStartSLOduration=2.348714014 podStartE2EDuration="2.348714014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:17.344096043 +0000 UTC m=+1449.844138344" watchObservedRunningTime="2026-10-29 22:11:17.348714014 +0000 UTC m=+1449.848756313"
Oct 29 22:11:17.385797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.385766    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"]
Oct 29 22:11:17.395078 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.395049    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a3c02e-5748cb5747-4dq8q"]
Oct 29 22:11:17.637018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:17.636921    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86f63896-629a-4327-a138-279748c035bd" path="/var/lib/kubelet/pods/86f63896-629a-4327-a138-279748c035bd/volumes"
Oct 29 22:11:20.744674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:20.728711    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"]
Oct 29 22:11:20.750678 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:20.750599    7082 cpu_manager.go:414] "containerMap: removing container" podUID="86f63896-629a-4327-a138-279748c035bd" containerName="payload-pre-processing"
Oct 29 22:11:20.760661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:20.756783    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f63896-629a-4327-a138-279748c035bd" containerName="payload-pre-processing"
Oct 29 22:11:20.794650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:20.774430    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f63896-629a-4327-a138-279748c035bd" containerName="payload-pre-processing"
Oct 29 22:11:20.994606 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:20.994543    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"
Oct 29 22:11:21.141649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.096197    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b80d5b-dockercfg-zgrlh\""
Oct 29 22:11:21.141649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.096338    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b80d5b\""
Oct 29 22:11:21.293651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.269013    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"]
Oct 29 22:11:21.302503 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.302418    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp\" (UID: \"ba49c3e6-27f2-4b68-b0a4-5c83c089395c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"
Oct 29 22:11:21.332424 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.332392    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"]
Oct 29 22:11:21.374740 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.371873    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt4n4\" (UniqueName: \"kubernetes.io/projected/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-kube-api-access-bt4n4\") pod \"payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp\" (UID: \"ba49c3e6-27f2-4b68-b0a4-5c83c089395c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"
Oct 29 22:11:21.384168 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.384136    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"]
Oct 29 22:11:21.384168 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.384256    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"
Oct 29 22:11:21.473960 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.473923    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt4n4\" (UniqueName: \"kubernetes.io/projected/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-kube-api-access-bt4n4\") pod \"payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp\" (UID: \"ba49c3e6-27f2-4b68-b0a4-5c83c089395c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"
Oct 29 22:11:21.474141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.474006    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp\" (UID: \"ba49c3e6-27f2-4b68-b0a4-5c83c089395c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"
Oct 29 22:11:21.474141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.474050    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnhx\" (UniqueName: \"kubernetes.io/projected/e175ca6e-a9dd-4596-9dbf-bba706737b2e-kube-api-access-dmnhx\") pod \"payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5\" (UID: \"e175ca6e-a9dd-4596-9dbf-bba706737b2e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"
Oct 29 22:11:21.474273 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.474203    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e175ca6e-a9dd-4596-9dbf-bba706737b2e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5\" (UID: \"e175ca6e-a9dd-4596-9dbf-bba706737b2e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"
Oct 29 22:11:21.474708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.474689    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp\" (UID: \"ba49c3e6-27f2-4b68-b0a4-5c83c089395c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"
Oct 29 22:11:21.499239 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.499217    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt4n4\" (UniqueName: \"kubernetes.io/projected/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-kube-api-access-bt4n4\") pod \"payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp\" (UID: \"ba49c3e6-27f2-4b68-b0a4-5c83c089395c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"
Oct 29 22:11:21.576392 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.576358    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnhx\" (UniqueName: \"kubernetes.io/projected/e175ca6e-a9dd-4596-9dbf-bba706737b2e-kube-api-access-dmnhx\") pod \"payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5\" (UID: \"e175ca6e-a9dd-4596-9dbf-bba706737b2e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"
Oct 29 22:11:21.576575 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.576516    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e175ca6e-a9dd-4596-9dbf-bba706737b2e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5\" (UID: \"e175ca6e-a9dd-4596-9dbf-bba706737b2e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"
Oct 29 22:11:21.577192 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.577173    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e175ca6e-a9dd-4596-9dbf-bba706737b2e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5\" (UID: \"e175ca6e-a9dd-4596-9dbf-bba706737b2e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"
Oct 29 22:11:21.599457 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.599426    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnhx\" (UniqueName: \"kubernetes.io/projected/e175ca6e-a9dd-4596-9dbf-bba706737b2e-kube-api-access-dmnhx\") pod \"payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5\" (UID: \"e175ca6e-a9dd-4596-9dbf-bba706737b2e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"
Oct 29 22:11:21.675650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.675574    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"
Oct 29 22:11:21.704184 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:21.704155    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"
Oct 29 22:11:22.001886 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:22.001857    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"]
Oct 29 22:11:22.002582 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:11:22.002550    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba49c3e6_27f2_4b68_b0a4_5c83c089395c.slice/crio-30a64f24738d1e0ad43127e331339581416aaf3727d79669640c447a84c554ae WatchSource:0}: Error finding container 30a64f24738d1e0ad43127e331339581416aaf3727d79669640c447a84c554ae: Status 404 returned error can't find the container with id 30a64f24738d1e0ad43127e331339581416aaf3727d79669640c447a84c554ae
Oct 29 22:11:22.045309 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:11:22.045278    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode175ca6e_a9dd_4596_9dbf_bba706737b2e.slice/crio-c01e3ad703481f9da97e9fb3b3f835a96e7879be2bcd68a6584e4cb46fce9418 WatchSource:0}: Error finding container c01e3ad703481f9da97e9fb3b3f835a96e7879be2bcd68a6584e4cb46fce9418: Status 404 returned error can't find the container with id c01e3ad703481f9da97e9fb3b3f835a96e7879be2bcd68a6584e4cb46fce9418
Oct 29 22:11:22.045409 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:22.045387    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"]
Oct 29 22:11:22.870337 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:22.870298    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp" event={"ID":"ba49c3e6-27f2-4b68-b0a4-5c83c089395c","Type":"ContainerStarted","Data":"392c768407dd8001f730fe1d55694a0ba1114b8f99c377a5dc7a9ac2f682f32a"}
Oct 29 22:11:22.870543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:22.870362    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp" event={"ID":"ba49c3e6-27f2-4b68-b0a4-5c83c089395c","Type":"ContainerStarted","Data":"30a64f24738d1e0ad43127e331339581416aaf3727d79669640c447a84c554ae"}
Oct 29 22:11:22.870882 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:22.870861    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"
Oct 29 22:11:22.874682 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:22.873072    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5" event={"ID":"e175ca6e-a9dd-4596-9dbf-bba706737b2e","Type":"ContainerStarted","Data":"18f51fbf625273184a657d420d27441e1580b71e0a23fc19236ee50c9c14e33f"}
Oct 29 22:11:22.874682 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:22.873117    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5" event={"ID":"e175ca6e-a9dd-4596-9dbf-bba706737b2e","Type":"ContainerStarted","Data":"c01e3ad703481f9da97e9fb3b3f835a96e7879be2bcd68a6584e4cb46fce9418"}
Oct 29 22:11:22.874682 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:22.873140    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"
Oct 29 22:11:22.880589 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:22.880565    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8c570_3e0f_40b7_9696_81b84b10e8eb.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:22.912988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:22.912932    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp" podStartSLOduration=2.912915053 podStartE2EDuration="2.912915053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:22.910526059 +0000 UTC m=+1455.410568359" watchObservedRunningTime="2026-10-29 22:11:22.912915053 +0000 UTC m=+1455.412957354"
Oct 29 22:11:22.955554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:22.955505    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5" podStartSLOduration=2.955492905 podStartE2EDuration="2.955492905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:22.950728926 +0000 UTC m=+1455.450771236" watchObservedRunningTime="2026-10-29 22:11:22.955492905 +0000 UTC m=+1455.455535231"
Oct 29 22:11:24.275737 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:24.275703    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8c570_3e0f_40b7_9696_81b84b10e8eb.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:26.038436 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:26.038404    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"]
Oct 29 22:11:26.049505 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:26.049478    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcade7ab-9w9wk"]
Oct 29 22:11:27.637534 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:27.637503    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08d562be-eb07-4d4b-be16-9cadd639f62d" path="/var/lib/kubelet/pods/08d562be-eb07-4d4b-be16-9cadd639f62d/volumes"
Oct 29 22:11:28.332863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:28.332833    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"
Oct 29 22:11:28.616705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:28.613061    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"]
Oct 29 22:11:28.650946 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:28.650871    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:11:28.790802 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:28.790765    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a3c02e-6rbjc"]
Oct 29 22:11:29.621277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:29.621240    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:11:29.635616 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:29.635586    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48e7dc6-2291-4349-84a1-40c6aa46a6df" path="/var/lib/kubelet/pods/b48e7dc6-2291-4349-84a1-40c6aa46a6df/volumes"
Oct 29 22:11:33.523708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.519861    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:33.575726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.556778    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"]
Oct 29 22:11:33.592662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.588597    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:33.753647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.727989    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:33.794704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.782782    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:33.813696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.805761    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6j8w\" (UniqueName: \"kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-kube-api-access-z6j8w\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:33.833655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.829592    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:33.833655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.829761    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:33.833655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.829820    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:33.833655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.830012    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:33.833655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.830086    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:33.833655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.830161    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:33.833655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.830366    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:34.014672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:33.979855    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:11:34.069832 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:34.063798    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"]
Oct 29 22:11:34.321875 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:34.321771    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:34.865667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:34.863330    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:11:34.907665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:34.877494    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": context deadline exceeded"
Oct 29 22:11:34.952782 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:34.952222    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.053652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.027031    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.317565 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.317521    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.348800 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.348739    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6j8w\" (UniqueName: \"kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-kube-api-access-z6j8w\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.426770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.424788    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.426770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.424905    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.426770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.424962    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.426770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.425138    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.426770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.425226    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.426770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.425303    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.426770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.425501    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.426770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.318125    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"
Oct 29 22:11:35.506652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.412846    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs" podUID="c32b079b-eba6-4e6f-8f79-f9c3cb86de99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.506652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.479254    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.506652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.366739    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.506652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.499322    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.506652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.499603    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.506652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.499903    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.506652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.500383    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"
Oct 29 22:11:35.506652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.504120    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.506652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.504226    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:11:35.506652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.505717    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:11:35.506652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.505752    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"]
Oct 29 22:11:35.535194 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.535162    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.552881 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.552843    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.741154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.673439    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.741154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.673518    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e69aa18-c109-4098-b08a-67695f24fa34-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.741154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.673578    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e69aa18-c109-4098-b08a-67695f24fa34-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.741154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.673657    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.741154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.673725    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.741154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.673828    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.741154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.673890    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk98d\" (UniqueName: \"kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-kube-api-access-qk98d\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.741154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.674038    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.741154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.674158    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.741154 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.725749    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6j8w\" (UniqueName: \"kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-kube-api-access-z6j8w\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.744662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.743184    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:35.897812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.897739    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.947658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.937303    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.966718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.965213    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.966718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.965291    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e69aa18-c109-4098-b08a-67695f24fa34-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.966718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.965358    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e69aa18-c109-4098-b08a-67695f24fa34-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.966718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.965418    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.966718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.965495    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.966718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.965595    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.966718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.965671    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk98d\" (UniqueName: \"kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-kube-api-access-qk98d\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.966718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.965774    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:35.966718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:35.965901    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:36.008708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.004820    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:36.040750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.024863    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:36.040977 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.025556    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e69aa18-c109-4098-b08a-67695f24fa34-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:36.041308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.041272    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:11:36.168213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.160500    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.308751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.306257    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:36.519654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.518701    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:36.519654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.519063    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:36.519913 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.519747    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:36.557017 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.556981    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk98d\" (UniqueName: \"kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-kube-api-access-qk98d\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:36.628709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.610642    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:36.649685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.631601    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:36.666241 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.666125    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:36.770701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:36.766558    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e69aa18-c109-4098-b08a-67695f24fa34-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:37.093840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:37.077517    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:37.096737 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:37.096700    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.311669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:37.203807    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.311669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:37.204369    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt" podUID="f069baec-59d9-4c9e-b5a2-c25261d01cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:37.502704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:37.499524    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs" podUID="c32b079b-eba6-4e6f-8f79-f9c3cb86de99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.557393 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:37.165611    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.790822 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:37.790717    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" podUID="4684c6fd-8ac7-4005-a20d-48396c0ab981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.894245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:37.894201    7082 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-wc4l2 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:37.918670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:37.894500    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wc4l2" podUID="4c2dd590-c639-42fe-8a02-4945b6403b3b" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.14:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:37.918975 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:37.782877    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:37.932168 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:37.911837    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.070735 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.054564    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.099185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.099139    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.222950 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.222820    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.515703 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:38.507040    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.738s"
Oct 29 22:11:38.515703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.507090    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"]
Oct 29 22:11:38.515703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.507263    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:11:38.515703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.507283    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"]
Oct 29 22:11:38.615189 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.615155    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:11:38.615474 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.615453    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:11:38.623724 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.623687    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"
Oct 29 22:11:38.645948 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.645914    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:11:38.646283 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.646267    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:11:38.671779 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.671731    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.819809 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.818447    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:11:38.819809 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.818502    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:11:38.979670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:38.978356    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:39.055936 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:39.055035    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:39.143711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:39.116929    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:39.237739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:39.234471    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:11:39.273521 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:39.273473    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp\" (UID: \"b25a95fc-3f00-488a-b750-1bb6383bcd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"
Oct 29 22:11:39.407376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:39.407268    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp\" (UID: \"b25a95fc-3f00-488a-b750-1bb6383bcd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"
Oct 29 22:11:39.407642 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:39.407592    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ktgd\" (UniqueName: \"kubernetes.io/projected/b25a95fc-3f00-488a-b750-1bb6383bcd95-kube-api-access-8ktgd\") pod \"maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp\" (UID: \"b25a95fc-3f00-488a-b750-1bb6383bcd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"
Oct 29 22:11:39.580894 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:39.580836    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp\" (UID: \"b25a95fc-3f00-488a-b750-1bb6383bcd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"
Oct 29 22:11:39.581298 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:39.581176    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ktgd\" (UniqueName: \"kubernetes.io/projected/b25a95fc-3f00-488a-b750-1bb6383bcd95-kube-api-access-8ktgd\") pod \"maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp\" (UID: \"b25a95fc-3f00-488a-b750-1bb6383bcd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"
Oct 29 22:11:39.605225 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:39.605180    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp\" (UID: \"b25a95fc-3f00-488a-b750-1bb6383bcd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"
Oct 29 22:11:39.672711 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:39.636581    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8c570_3e0f_40b7_9696_81b84b10e8eb.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:39.672711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:39.656310    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp\" (UID: \"b25a95fc-3f00-488a-b750-1bb6383bcd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"
Oct 29 22:11:39.683156 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:39.683118    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.067s"
Oct 29 22:11:39.779722 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:39.754458    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8c570_3e0f_40b7_9696_81b84b10e8eb.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:39.987343 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:39.987239    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded"
Oct 29 22:11:40.074597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:39.989741    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:11:40.135662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:40.130897    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:40.207185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:40.139270    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:40.219872 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:40.218004    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:11:40.299692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:40.299595    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ktgd\" (UniqueName: \"kubernetes.io/projected/b25a95fc-3f00-488a-b750-1bb6383bcd95-kube-api-access-8ktgd\") pod \"maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp\" (UID: \"b25a95fc-3f00-488a-b750-1bb6383bcd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"
Oct 29 22:11:40.511483 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:40.482030    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:11:40.657205 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:40.657170    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp\" (UID: \"b25a95fc-3f00-488a-b750-1bb6383bcd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"
Oct 29 22:11:40.677762 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:40.677722    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"
Oct 29 22:11:41.134599 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:41.096587    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.362s"
Oct 29 22:11:41.134599 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:41.096758    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"]
Oct 29 22:11:41.134599 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:41.096841    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"]
Oct 29 22:11:41.134599 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:41.096885    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"]
Oct 29 22:11:41.134599 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:41.097129    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy" containerID="cri-o://ff4adb076d831437ba08c8a7653fda6c51e2b9cac16123fa4f5fbd656878318f" gracePeriod=30
Oct 29 22:11:41.149183 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:41.145210    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy" containerID="cri-o://8d4f90c73ffefa4754a913a590dfe478c7d1d13715131a26c85853fca9f258d3" gracePeriod=30
Oct 29 22:11:41.161652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:41.155752    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.221885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:41.217780    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.221885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:41.218692    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.360653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:41.357204    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:11:41.530446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:41.530386    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.697675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:41.692590    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"]
Oct 29 22:11:41.807053 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:41.769809    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy" containerID="cri-o://2b1a6483ffc3e7338970e087437f16f0ba436d7946c8b5f1eda54d13a8f21884" gracePeriod=30
Oct 29 22:11:42.032653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:42.014301    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt" podUID="f069baec-59d9-4c9e-b5a2-c25261d01cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:11:42.281708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:42.268464    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.720652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:42.295420    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.763468 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:42.295460    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:42.780653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:42.580726    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Oct 29 22:11:42.780653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:42.617591    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:11:42.807657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:42.797230    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:11:42.807657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:42.797339    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:11:42.807657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:42.797384    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:11:42.836956 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:42.778173    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" podUID="0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:11:43.027049 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.026943    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:11:43.093307 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.087891    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" podUID="4684c6fd-8ac7-4005-a20d-48396c0ab981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.094876 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.094826    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:43.135734 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.135692    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.137491 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:43.121471    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.303s"
Oct 29 22:11:43.137812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.137792    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"]
Oct 29 22:11:43.138155 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.138128    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerName="istio-proxy" containerID="cri-o://c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e" gracePeriod=30
Oct 29 22:11:43.164127 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.164061    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:11:43.185961 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.185907    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:11:43.186308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.186275    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:11:43.564678 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.547703    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:43.564678 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.547770    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.618590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.600735    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:43.618590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.600798    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.732189 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.684076    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.858787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.774839    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.889654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.774877    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:11:43.891089 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.774899    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:11:43.903200 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:43.783944    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:44.029705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:44.029597    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:44.030695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:44.030659    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.031865 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:44.031837    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.360275 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:44.360232    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" event={"ID":"2e69aa18-c109-4098-b08a-67695f24fa34","Type":"ContainerStarted","Data":"71b18c7b4a48218cd7acf7a51efa852d5cd3a189d55ceea73cc1c25c4f58bd26"}
Oct 29 22:11:44.360576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:44.360551    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" event={"ID":"ac2febe4-d73f-42e3-bf74-ce53ba139b78","Type":"ContainerStarted","Data":"082a535a433e682200cba328fce42ce8e62e37a658eedc86c828a12b81f611de"}
Oct 29 22:11:45.044552 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:45.036430    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:45.203748 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:45.134158    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": context deadline exceeded"
Oct 29 22:11:45.290794 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:45.290713    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:45.398001 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:45.291705    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:45.398001 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:45.291961    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:11:45.769713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:45.769661    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:11:46.062706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:46.041227    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:46.298663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:46.294881    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:46.298663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:46.295293    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:46.298663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:46.295585    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:46.477654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:46.459550    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"]
Oct 29 22:11:46.827820 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:46.806895    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"]
Oct 29 22:11:46.847660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:46.839749    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:47.087743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:47.087277    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:11:47.118520 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:47.117683    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"]
Oct 29 22:11:47.377703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:47.377558    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:47.392663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:47.378930    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:11:47.392663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:47.379819    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:11:47.556680 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:47.556615    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-a9def7-6ql6b"]
Oct 29 22:11:47.868788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:47.845838    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:47.970492 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:47.862999    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:11:48.165501 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.165195    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:11:48.179280 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.178852    7082 generic.go:356] "Generic (PLEG): container finished" podID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerID="ff4adb076d831437ba08c8a7653fda6c51e2b9cac16123fa4f5fbd656878318f" exitCode=0
Oct 29 22:11:48.182471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.182442    7082 generic.go:356] "Generic (PLEG): container finished" podID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerID="8d4f90c73ffefa4754a913a590dfe478c7d1d13715131a26c85853fca9f258d3" exitCode=0
Oct 29 22:11:48.379169 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.378942    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b469f3-9bf7-4da9-9c75-98b44db7c344" path="/var/lib/kubelet/pods/35b469f3-9bf7-4da9-9c75-98b44db7c344/volumes"
Oct 29 22:11:48.380037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.380003    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"]
Oct 29 22:11:48.380240 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.380219    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" event={"ID":"3379fba3-ce1e-4577-a370-ab3d675b4c4b","Type":"ContainerDied","Data":"ff4adb076d831437ba08c8a7653fda6c51e2b9cac16123fa4f5fbd656878318f"}
Oct 29 22:11:48.380421 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.380399    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" event={"ID":"85ff3c7f-c721-44e3-b83e-b214b93fc2b8","Type":"ContainerDied","Data":"8d4f90c73ffefa4754a913a590dfe478c7d1d13715131a26c85853fca9f258d3"}
Oct 29 22:11:48.380554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.380539    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" event={"ID":"2e69aa18-c109-4098-b08a-67695f24fa34","Type":"ContainerStarted","Data":"d2edb2bf6d0af4a306c232ab7eb5ccc9e94c6a6854bec0f54b5233e6b40ce579"}
Oct 29 22:11:48.380761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.380740    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" event={"ID":"ac2febe4-d73f-42e3-bf74-ce53ba139b78","Type":"ContainerStarted","Data":"82a5b751361e0a22c78cefb1ab9fd7766600fab043101418f6bb9810dd694eea"}
Oct 29 22:11:48.381019 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.380997    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp" event={"ID":"b25a95fc-3f00-488a-b750-1bb6383bcd95","Type":"ContainerStarted","Data":"8e2d583c9b38dff9319f0bb981b7029f388aaa3db6fa4dc2ba355ff8b158ecd7"}
Oct 29 22:11:48.381238 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.381220    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp" event={"ID":"b25a95fc-3f00-488a-b750-1bb6383bcd95","Type":"ContainerStarted","Data":"3268d64e7056c78da0c5c2ce899725266e6f5ff5079ac4d7a10271f74f184afb"}
Oct 29 22:11:48.381357 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.379261    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:48.382134 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.380061    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:48.474887 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.447335    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:48.484666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.476466    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" podStartSLOduration=15.476443229000001 podStartE2EDuration="15.476443229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:48.449207383 +0000 UTC m=+1480.949249690" watchObservedRunningTime="2026-10-29 22:11:48.476443229 +0000 UTC m=+1480.976485528"
Oct 29 22:11:48.574880 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.517014    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": read tcp 10.134.0.2:46258->10.134.0.98:15021: read: connection reset by peer" start-of-body=
Oct 29 22:11:48.574880 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.517074    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": read tcp 10.134.0.2:46258->10.134.0.98:15021: read: connection reset by peer"
Oct 29 22:11:48.703018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.702983    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:11:48.703297 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.703266    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 29 22:11:49.010866 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.979877    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:49.010866 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:48.996198    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:11:49.179109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.177595    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:49.252662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.252604    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 22:11:49.252977 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.252946    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 22:11:49.329355 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.311097    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:49.329355 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.311413    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:11:49.329355 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.311469    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 22:11:49.447773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.447367    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:49.448019 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.447768    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:11:49.465130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.464890    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:49.597708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.555932    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" podStartSLOduration=16.555905406 podStartE2EDuration="16.555905406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:48.598978213 +0000 UTC m=+1481.099020514" watchObservedRunningTime="2026-10-29 22:11:49.555905406 +0000 UTC m=+1482.055947708"
Oct 29 22:11:50.033711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.032770    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded"
Oct 29 22:11:50.033711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.668098    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-zqhtx" podUID="4684c6fd-8ac7-4005-a20d-48396c0ab981" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:11:50.066670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.685024    7082 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:50.066670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.034726    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.066670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.782996    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 22:11:50.066670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.034795    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 22:11:50.066670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.042862    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt" podUID="f069baec-59d9-4c9e-b5a2-c25261d01cf4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.066670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.045914    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:11:50.066670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.059120    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp" podStartSLOduration=16.059102153 podStartE2EDuration="16.059102153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:48.728515995 +0000 UTC m=+1481.228558331" watchObservedRunningTime="2026-10-29 22:11:50.059102153 +0000 UTC m=+1482.559144453"
Oct 29 22:11:50.066670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.064525    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.095006 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:49.581861    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" podUID="0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.148843 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.142949    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:11:50.148843 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.143005    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 22:11:50.197519 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.197472    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:11:50.415652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.358253    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-kube-api-access-5m6h2\" (UniqueName: \"kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-kube-api-access-5m6h2\") pod \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") "
Oct 29 22:11:50.451663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.448841    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-envoy\") pod \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") "
Oct 29 22:11:50.686025 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.685940    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-data\") pod \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") "
Oct 29 22:11:50.703096 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.703056    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-socket\") pod \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") "
Oct 29 22:11:50.703445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.703421    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-certs\") pod \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") "
Oct 29 22:11:50.703691 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.703673    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-token\" (UniqueName: \"kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-token\") pod \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") "
Oct 29 22:11:50.748667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.736117    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-podinfo\") pod \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") "
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.752872    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-credential-socket\") pod \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") "
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.752996    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istiod-ca-cert\") pod \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\" (UID: \"85ff3c7f-c721-44e3-b83e-b214b93fc2b8\") "
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.445811    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-kube-api-access-5m6h2" pod "85ff3c7f-c721-44e3-b83e-b214b93fc2b8" (UID: "85ff3c7f-c721-44e3-b83e-b214b93fc2b8"). InnerVolumeSpecName "kube-api-access-5m6h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.689201    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-data" pod "85ff3c7f-c721-44e3-b83e-b214b93fc2b8" (UID: "85ff3c7f-c721-44e3-b83e-b214b93fc2b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.722126    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-socket" pod "85ff3c7f-c721-44e3-b83e-b214b93fc2b8" (UID: "85ff3c7f-c721-44e3-b83e-b214b93fc2b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.722311    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-certs" pod "85ff3c7f-c721-44e3-b83e-b214b93fc2b8" (UID: "85ff3c7f-c721-44e3-b83e-b214b93fc2b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.461075    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.554930    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.755245    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-credential-socket" pod "85ff3c7f-c721-44e3-b83e-b214b93fc2b8" (UID: "85ff3c7f-c721-44e3-b83e-b214b93fc2b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.683519    7082 generic.go:356] "Generic (PLEG): container finished" podID="ddab3d72-73bf-4033-be01-6177dddff66a" containerID="2b1a6483ffc3e7338970e087437f16f0ba436d7946c8b5f1eda54d13a8f21884" exitCode=0
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.755764    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istiod-ca-cert" pod "85ff3c7f-c721-44e3-b83e-b214b93fc2b8" (UID: "85ff3c7f-c721-44e3-b83e-b214b93fc2b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.448325    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.448357    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.442578    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.756212    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6" event={"ID":"85ff3c7f-c721-44e3-b83e-b214b93fc2b8","Type":"ContainerDied","Data":"c0749db461789df6178c8dcecac6ed1b453d1e31c90c1c415bc3088b0ce07556"}
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.756246    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" event={"ID":"ddab3d72-73bf-4033-be01-6177dddff66a","Type":"ContainerDied","Data":"2b1a6483ffc3e7338970e087437f16f0ba436d7946c8b5f1eda54d13a8f21884"}
Oct 29 22:11:50.769667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.756281    7082 scope.go:122] "RemoveContainer" containerID="8d4f90c73ffefa4754a913a590dfe478c7d1d13715131a26c85853fca9f258d3"
Oct 29 22:11:50.831804 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.831765    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 22:11:50.832117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.832065    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 22:11:50.840657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.840594    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-envoy" pod "85ff3c7f-c721-44e3-b83e-b214b93fc2b8" (UID: "85ff3c7f-c721-44e3-b83e-b214b93fc2b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:50.952516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.869181    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-podinfo" pod "85ff3c7f-c721-44e3-b83e-b214b93fc2b8" (UID: "85ff3c7f-c721-44e3-b83e-b214b93fc2b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:50.952516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.950264    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:50.952516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.950295    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:50.952516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.950310    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:50.952516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.950324    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m6h2\" (UniqueName: \"kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-kube-api-access-5m6h2\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:50.952516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.950338    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:50.952516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.950357    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:50.952516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.950370    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:50.952516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.950382    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:50.982008 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:50.967808    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-token" pod "85ff3c7f-c721-44e3-b83e-b214b93fc2b8" (UID: "85ff3c7f-c721-44e3-b83e-b214b93fc2b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:51.103920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.103880    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:11:51.104219 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.104173    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 22:11:51.154656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.150465    7082 generic.go:356] "Generic (PLEG): container finished" podID="b25a95fc-3f00-488a-b750-1bb6383bcd95" containerID="8e2d583c9b38dff9319f0bb981b7029f388aaa3db6fa4dc2ba355ff8b158ecd7" exitCode=0
Oct 29 22:11:51.165782 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.155040    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp" event={"ID":"b25a95fc-3f00-488a-b750-1bb6383bcd95","Type":"ContainerDied","Data":"8e2d583c9b38dff9319f0bb981b7029f388aaa3db6fa4dc2ba355ff8b158ecd7"}
Oct 29 22:11:51.249131 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.241460    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ff3c7f-c721-44e3-b83e-b214b93fc2b8-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:51.249131 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.242041    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:51.604932 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.604872    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:51.758744 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.758691    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:51.759680 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.759620    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:51.760516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.760473    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:11:51.842121 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.842024    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 22:11:51.842407 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.842356    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 22:11:51.961750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.961712    7082 patch_prober.go:29] interesting pod/thanos-querier-f96f5cd85-t7ndb container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded" start-of-body=
Oct 29 22:11:51.968744 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:51.968698    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-f96f5cd85-t7ndb" podUID="b96d72b2-bc45-4962-b529-f08155ad0215" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.13:9091/-/ready\": context deadline exceeded"
Oct 29 22:11:52.220960 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.220871    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:11:52.221218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.221188    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 22:11:52.298706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.259429    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Oct 29 22:11:52.378820 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.301720    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs" podUID="c32b079b-eba6-4e6f-8f79-f9c3cb86de99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:52.379723 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.367422    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:11:52.400409 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.400380    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 22:11:52.477779 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.477688    7082 generic.go:356] "Generic (PLEG): container finished" podID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerID="c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e" exitCode=0
Oct 29 22:11:52.598931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.598889    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-credential-socket\") pod \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") "
Oct 29 22:11:52.599475 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.599444    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-token\" (UniqueName: \"kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-token\") pod \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") "
Oct 29 22:11:52.599702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.599684    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istiod-ca-cert\") pod \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") "
Oct 29 22:11:52.599895 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.599864    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-data\") pod \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") "
Oct 29 22:11:52.600097 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.600062    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-podinfo\") pod \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") "
Oct 29 22:11:52.600304 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.600289    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-socket\") pod \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") "
Oct 29 22:11:52.600484 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.600454    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-kube-api-access-th494\" (UniqueName: \"kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-kube-api-access-th494\") pod \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") "
Oct 29 22:11:52.600666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.600649    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-envoy\") pod \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") "
Oct 29 22:11:52.600872 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.600855    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-certs\") pod \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\" (UID: \"3379fba3-ce1e-4577-a370-ab3d675b4c4b\") "
Oct 29 22:11:52.612276 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.612245    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-socket" pod "3379fba3-ce1e-4577-a370-ab3d675b4c4b" (UID: "3379fba3-ce1e-4577-a370-ab3d675b4c4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:52.639536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.639497    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-credential-socket" pod "3379fba3-ce1e-4577-a370-ab3d675b4c4b" (UID: "3379fba3-ce1e-4577-a370-ab3d675b4c4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:52.641401 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.641376    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:52.641799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.641781    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:52.665288 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.665245    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-data" pod "3379fba3-ce1e-4577-a370-ab3d675b4c4b" (UID: "3379fba3-ce1e-4577-a370-ab3d675b4c4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:52.801562 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.801509    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-certs" pod "3379fba3-ce1e-4577-a370-ab3d675b4c4b" (UID: "3379fba3-ce1e-4577-a370-ab3d675b4c4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:52.803173 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.803146    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 22:11:52.803407 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.803373    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 22:11:52.825343 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.825309    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istiod-ca-cert" pod "3379fba3-ce1e-4577-a370-ab3d675b4c4b" (UID: "3379fba3-ce1e-4577-a370-ab3d675b4c4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:52.848653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.830046    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:52.848940 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.848918    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:52.849046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.849035    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:52.955876 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.913058    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:11:52.955876 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.932714    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:52.969799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:52.963432    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:53.125923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.125833    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 22:11:53.163640 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.152916    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-token\" (UniqueName: \"kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-token\") pod \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") "
Oct 29 22:11:53.163640 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.152967    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-certs\") pod \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") "
Oct 29 22:11:53.163640 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.153100    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-credential-socket\") pod \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") "
Oct 29 22:11:53.163640 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.153229    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-envoy\") pod \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") "
Oct 29 22:11:53.163640 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.153274    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-kube-api-access-npmkq\" (UniqueName: \"kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-kube-api-access-npmkq\") pod \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") "
Oct 29 22:11:53.163640 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.155464    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-socket\") pod \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") "
Oct 29 22:11:53.163640 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.155542    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istiod-ca-cert\") pod \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") "
Oct 29 22:11:53.163640 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.155572    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-data\") pod \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") "
Oct 29 22:11:53.163640 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.155742    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-podinfo\") pod \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\" (UID: \"58e47b26-da7b-4e93-aa45-16fc7cca65d0\") "
Oct 29 22:11:53.246354 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.246312    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-credential-socket" pod "58e47b26-da7b-4e93-aa45-16fc7cca65d0" (UID: "58e47b26-da7b-4e93-aa45-16fc7cca65d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:53.436361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.285846    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-certs" pod "58e47b26-da7b-4e93-aa45-16fc7cca65d0" (UID: "58e47b26-da7b-4e93-aa45-16fc7cca65d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:53.436361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.309979    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-podinfo" pod "58e47b26-da7b-4e93-aa45-16fc7cca65d0" (UID: "58e47b26-da7b-4e93-aa45-16fc7cca65d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:53.436361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.410880    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istiod-ca-cert" pod "58e47b26-da7b-4e93-aa45-16fc7cca65d0" (UID: "58e47b26-da7b-4e93-aa45-16fc7cca65d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:53.436361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.411170    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-socket" pod "58e47b26-da7b-4e93-aa45-16fc7cca65d0" (UID: "58e47b26-da7b-4e93-aa45-16fc7cca65d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:53.436361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.411250    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-envoy" pod "3379fba3-ce1e-4577-a370-ab3d675b4c4b" (UID: "3379fba3-ce1e-4577-a370-ab3d675b4c4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:53.436361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.411317    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:11:53.436361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.411367    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 22:11:53.436361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.411730    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:53.436361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.425536    7082 scope.go:122] "RemoveContainer" containerID="8d4f90c73ffefa4754a913a590dfe478c7d1d13715131a26c85853fca9f258d3"
Oct 29 22:11:53.464818 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.442856    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-data" pod "58e47b26-da7b-4e93-aa45-16fc7cca65d0" (UID: "58e47b26-da7b-4e93-aa45-16fc7cca65d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:53.479651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.470063    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:53.479651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.470102    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:53.479651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.470117    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:53.479651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.470128    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:53.479651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.470139    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:53.479651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.470150    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:53.479651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.470162    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:53.527790 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.490812    7082 scope.go:122] "RemoveContainer" containerID="49b3ee64fd31e713c9cdb45735adec0534d19db33d957d3c3794824a8536c8a5"
Oct 29 22:11:53.527790 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.492231    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:53.528552 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.528509    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:53.587670 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:53.567931    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.912s"
Oct 29 22:11:53.587670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.567977    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:53.587670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.568135    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:53.587670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.568162    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"]
Oct 29 22:11:53.587670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.568187    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-xjrf6"]
Oct 29 22:11:53.587670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.568210    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" event={"ID":"58e47b26-da7b-4e93-aa45-16fc7cca65d0","Type":"ContainerDied","Data":"c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e"}
Oct 29 22:11:53.587670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.568274    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:11:53.587670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.568297    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"]
Oct 29 22:11:53.587670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.568362    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:11:53.670653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.648495    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"]
Oct 29 22:11:53.670989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.670962    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"]
Oct 29 22:11:53.693704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.671311    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy" containerID="cri-o://9e600ad115f340536d61e56ed2d7473645968cf243a1a9a3e895485c4874b8bf" gracePeriod=30
Oct 29 22:11:53.729962 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.697002    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy" containerID="cri-o://7aad63497c30527e83474c40b35e6cbba76cac0a29d3eaa5bf4b095eee43dd27" gracePeriod=30
Oct 29 22:11:53.729962 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.697598    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy" containerID="cri-o://dd2cb7245a324b8e286f9e9126733b31ef6ab17ef737554f47ade48e489de7c5" gracePeriod=30
Oct 29 22:11:53.785827 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.782101    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-token" pod "3379fba3-ce1e-4577-a370-ab3d675b4c4b" (UID: "3379fba3-ce1e-4577-a370-ab3d675b4c4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:53.897871 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:53.818601    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4f90c73ffefa4754a913a590dfe478c7d1d13715131a26c85853fca9f258d3\": container with ID starting with 8d4f90c73ffefa4754a913a590dfe478c7d1d13715131a26c85853fca9f258d3 not found: ID does not exist" containerID="8d4f90c73ffefa4754a913a590dfe478c7d1d13715131a26c85853fca9f258d3"
Oct 29 22:11:53.897871 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:53.862518    7082 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"8d4f90c73ffefa4754a913a590dfe478c7d1d13715131a26c85853fca9f258d3\": rpc error: code = NotFound desc = could not find container \"8d4f90c73ffefa4754a913a590dfe478c7d1d13715131a26c85853fca9f258d3\": container with ID starting with 8d4f90c73ffefa4754a913a590dfe478c7d1d13715131a26c85853fca9f258d3 not found: ID does not exist" containerID="8d4f90c73ffefa4754a913a590dfe478c7d1d13715131a26c85853fca9f258d3"
Oct 29 22:11:53.897871 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.862559    7082 scope.go:122] "RemoveContainer" containerID="ff4adb076d831437ba08c8a7653fda6c51e2b9cac16123fa4f5fbd656878318f"
Oct 29 22:11:53.951595 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.951508    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 22:11:54.064861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.005231    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 22:11:54.064861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:53.963771    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:54.064861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.006215    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-token" pod "58e47b26-da7b-4e93-aa45-16fc7cca65d0" (UID: "58e47b26-da7b-4e93-aa45-16fc7cca65d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:54.064861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.038889    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:11:54.064861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.040990    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-podinfo" pod "3379fba3-ce1e-4577-a370-ab3d675b4c4b" (UID: "3379fba3-ce1e-4577-a370-ab3d675b4c4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:54.064861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.044168    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-envoy" pod "58e47b26-da7b-4e93-aa45-16fc7cca65d0" (UID: "58e47b26-da7b-4e93-aa45-16fc7cca65d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:54.135274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.135224    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:54.135580 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.135561    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3379fba3-ce1e-4577-a370-ab3d675b4c4b-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:54.135781 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.135767    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e47b26-da7b-4e93-aa45-16fc7cca65d0-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:54.136052 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.136019    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-kube-api-access-th494" pod "3379fba3-ce1e-4577-a370-ab3d675b4c4b" (UID: "3379fba3-ce1e-4577-a370-ab3d675b4c4b"). InnerVolumeSpecName "kube-api-access-th494". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:54.266705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.248443    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th494\" (UniqueName: \"kubernetes.io/projected/3379fba3-ce1e-4577-a370-ab3d675b4c4b-kube-api-access-th494\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:54.355908 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.321464    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:11:54.355908 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.321771    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:54.548700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.513882    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 22:11:54.697055 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.697013    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:11:54.719342 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.719300    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 22:11:54.885656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.883174    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-istio-token\" (UniqueName: \"kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-istio-token\") pod \"ddab3d72-73bf-4033-be01-6177dddff66a\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") "
Oct 29 22:11:54.885656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.883275    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-kube-api-access-wpqxq\" (UniqueName: \"kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-kube-api-access-wpqxq\") pod \"ddab3d72-73bf-4033-be01-6177dddff66a\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") "
Oct 29 22:11:54.885656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.883526    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-data\") pod \"ddab3d72-73bf-4033-be01-6177dddff66a\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") "
Oct 29 22:11:54.885656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.883594    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-socket\") pod \"ddab3d72-73bf-4033-be01-6177dddff66a\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") "
Oct 29 22:11:54.885656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.883732    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-envoy\") pod \"ddab3d72-73bf-4033-be01-6177dddff66a\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") "
Oct 29 22:11:54.885656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.883801    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddab3d72-73bf-4033-be01-6177dddff66a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddab3d72-73bf-4033-be01-6177dddff66a-istio-podinfo\") pod \"ddab3d72-73bf-4033-be01-6177dddff66a\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") "
Oct 29 22:11:54.885656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.883858    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-credential-socket\") pod \"ddab3d72-73bf-4033-be01-6177dddff66a\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") "
Oct 29 22:11:54.885656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.883903    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddab3d72-73bf-4033-be01-6177dddff66a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddab3d72-73bf-4033-be01-6177dddff66a-istiod-ca-cert\") pod \"ddab3d72-73bf-4033-be01-6177dddff66a\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") "
Oct 29 22:11:54.885656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:54.884343    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-certs\") pod \"ddab3d72-73bf-4033-be01-6177dddff66a\" (UID: \"ddab3d72-73bf-4033-be01-6177dddff66a\") "
Oct 29 22:11:55.104916 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.104882    7082 scope.go:122] "RemoveContainer" containerID="821bce1497ebe7c190a7a37a145e84d9b3c25e2bb9b3112f77f5f1991956201d"
Oct 29 22:11:55.163647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.120054    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:11:55.328715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.325067    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:55.663742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.162462    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": context deadline exceeded"
Oct 29 22:11:55.663742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.169537    7082 scope.go:122] "RemoveContainer" containerID="c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e"
Oct 29 22:11:55.663742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.176235    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddab3d72-73bf-4033-be01-6177dddff66a-istiod-ca-cert" pod "ddab3d72-73bf-4033-be01-6177dddff66a" (UID: "ddab3d72-73bf-4033-be01-6177dddff66a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:55.663742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.176808    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-certs" pod "ddab3d72-73bf-4033-be01-6177dddff66a" (UID: "ddab3d72-73bf-4033-be01-6177dddff66a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.663742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.179038    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-kube-api-access-npmkq" pod "58e47b26-da7b-4e93-aa45-16fc7cca65d0" (UID: "58e47b26-da7b-4e93-aa45-16fc7cca65d0"). InnerVolumeSpecName "kube-api-access-npmkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:55.663742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.284947    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 22:11:55.663742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.662053    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 22:11:55.663742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.346173    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-credential-socket" pod "ddab3d72-73bf-4033-be01-6177dddff66a" (UID: "ddab3d72-73bf-4033-be01-6177dddff66a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.663742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.374053    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-istio-token" pod "ddab3d72-73bf-4033-be01-6177dddff66a" (UID: "ddab3d72-73bf-4033-be01-6177dddff66a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:55.668299 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.668267    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-data" pod "ddab3d72-73bf-4033-be01-6177dddff66a" (UID: "ddab3d72-73bf-4033-be01-6177dddff66a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.748656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.712408    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.755661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.749051    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddab3d72-73bf-4033-be01-6177dddff66a-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.776928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.776905    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.859751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.858414    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.859751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.858451    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npmkq\" (UniqueName: \"kubernetes.io/projected/58e47b26-da7b-4e93-aa45-16fc7cca65d0-kube-api-access-npmkq\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.859751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.858476    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.859751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.838053    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"
Oct 29 22:11:55.868855 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.764677    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"
Oct 29 22:11:55.868855 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:55.857947    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8c570_3e0f_40b7_9696_81b84b10e8eb.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:55.871206 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.871182    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"
Oct 29 22:11:55.878559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.872928    7082 scope.go:122] "RemoveContainer" containerID="49b3ee64fd31e713c9cdb45735adec0534d19db33d957d3c3794824a8536c8a5"
Oct 29 22:11:55.878559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.163905    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:55.880554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.726819    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-kube-api-access-wpqxq" pod "ddab3d72-73bf-4033-be01-6177dddff66a" (UID: "ddab3d72-73bf-4033-be01-6177dddff66a"). InnerVolumeSpecName "kube-api-access-wpqxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:55.880554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.777597    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 22:11:55.880554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.878882    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 22:11:55.880554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.814692    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddab3d72-73bf-4033-be01-6177dddff66a-istio-podinfo" pod "ddab3d72-73bf-4033-be01-6177dddff66a" (UID: "ddab3d72-73bf-4033-be01-6177dddff66a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:55.880554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.815301    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:11:55.880554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.878967    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 22:11:55.880554 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:55.851908    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8c570_3e0f_40b7_9696_81b84b10e8eb.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:55.880999 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:55.880765    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b3ee64fd31e713c9cdb45735adec0534d19db33d957d3c3794824a8536c8a5\": container with ID starting with 49b3ee64fd31e713c9cdb45735adec0534d19db33d957d3c3794824a8536c8a5 not found: ID does not exist" containerID="49b3ee64fd31e713c9cdb45735adec0534d19db33d957d3c3794824a8536c8a5"
Oct 29 22:11:55.880999 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:55.880803    7082 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"49b3ee64fd31e713c9cdb45735adec0534d19db33d957d3c3794824a8536c8a5\": rpc error: code = NotFound desc = could not find container \"49b3ee64fd31e713c9cdb45735adec0534d19db33d957d3c3794824a8536c8a5\": container with ID starting with 49b3ee64fd31e713c9cdb45735adec0534d19db33d957d3c3794824a8536c8a5 not found: ID does not exist" containerID="49b3ee64fd31e713c9cdb45735adec0534d19db33d957d3c3794824a8536c8a5"
Oct 29 22:11:55.880999 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.880835    7082 scope.go:122] "RemoveContainer" containerID="2b1a6483ffc3e7338970e087437f16f0ba436d7946c8b5f1eda54d13a8f21884"
Oct 29 22:11:55.893652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.891698    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-socket" pod "ddab3d72-73bf-4033-be01-6177dddff66a" (UID: "ddab3d72-73bf-4033-be01-6177dddff66a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.915849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.914376    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-envoy" pod "ddab3d72-73bf-4033-be01-6177dddff66a" (UID: "ddab3d72-73bf-4033-be01-6177dddff66a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.956872 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.956831    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" path="/var/lib/kubelet/pods/85ff3c7f-c721-44e3-b83e-b214b93fc2b8/volumes"
Oct 29 22:11:55.964360 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:55.964336    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.253s"
Oct 29 22:11:55.964557 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.964534    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f" event={"ID":"ddab3d72-73bf-4033-be01-6177dddff66a","Type":"ContainerDied","Data":"2dc598583b9ecc726b42d1a697bd0fd733dfad52d85c8519b0b97c35e6701262"}
Oct 29 22:11:55.964758 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.964739    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm" event={"ID":"58e47b26-da7b-4e93-aa45-16fc7cca65d0","Type":"ContainerDied","Data":"13ea18c085d67134950166cffeed3bc7e237ae97054ccffb11d7c2980dfbb2d9"}
Oct 29 22:11:55.964930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.964911    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns" event={"ID":"3379fba3-ce1e-4577-a370-ab3d675b4c4b","Type":"ContainerDied","Data":"482bed1e652cf2c0decc67af645ef8a9b3d6610d3abeec83acd0cab1347ad776"}
Oct 29 22:11:55.965065 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.965050    7082 scope.go:122] "RemoveContainer" containerID="2b1a6483ffc3e7338970e087437f16f0ba436d7946c8b5f1eda54d13a8f21884"
Oct 29 22:11:55.969610 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.969584    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.969717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.969615    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddab3d72-73bf-4033-be01-6177dddff66a-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.969717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.969656    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpqxq\" (UniqueName: \"kubernetes.io/projected/ddab3d72-73bf-4033-be01-6177dddff66a-kube-api-access-wpqxq\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.969717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:55.969673    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddab3d72-73bf-4033-be01-6177dddff66a-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:56.059968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.059363    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:11:56.059968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.059452    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:11:56.059968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.059522    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:11:56.059968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.059562    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:11:56.059968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.059610    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:11:56.059968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.059668    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:11:56.278461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.273283    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:11:56.278461 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.273342    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 22:11:56.368005 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.367877    7082 scope.go:122] "RemoveContainer" containerID="c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e"
Oct 29 22:11:56.371843 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:56.371543    7082 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f_openshift-ingress_ddab3d72-73bf-4033-be01-6177dddff66a_1 in pod sandbox 2dc598583b9ecc726b42d1a697bd0fd733dfad52d85c8519b0b97c35e6701262 from index: no such id: '2b1a6483ffc3e7338970e087437f16f0ba436d7946c8b5f1eda54d13a8f21884'" containerID="2b1a6483ffc3e7338970e087437f16f0ba436d7946c8b5f1eda54d13a8f21884"
Oct 29 22:11:56.371989 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:56.371861    7082 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f_openshift-ingress_ddab3d72-73bf-4033-be01-6177dddff66a_1 in pod sandbox 2dc598583b9ecc726b42d1a697bd0fd733dfad52d85c8519b0b97c35e6701262 from index: no such id: '2b1a6483ffc3e7338970e087437f16f0ba436d7946c8b5f1eda54d13a8f21884'" containerID="2b1a6483ffc3e7338970e087437f16f0ba436d7946c8b5f1eda54d13a8f21884"
Oct 29 22:11:56.371989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.371903    7082 scope.go:122] "RemoveContainer" containerID="c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e"
Oct 29 22:11:56.414937 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:56.403381    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e\": container with ID starting with c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e not found: ID does not exist" containerID="c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e"
Oct 29 22:11:56.414937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.403434    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e"} err="failed to get container status \"c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e\": rpc error: code = NotFound desc = could not find container \"c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e\": container with ID starting with c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e not found: ID does not exist"
Oct 29 22:11:56.416913 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:56.416484    7082 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e\": rpc error: code = NotFound desc = could not find container \"c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e\": container with ID starting with c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e not found: ID does not exist" containerID="c498c480d3d66625abde95ebf65acd4b0aeaaa2f234007c0011b1f5c12c6ca0e"
Oct 29 22:11:56.637654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.617270    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"]
Oct 29 22:11:56.637654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.631889    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-cwfbm"]
Oct 29 22:11:56.763409 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.763361    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"]
Oct 29 22:11:56.766550 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.766516    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 29 22:11:56.766704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.766573    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 29 22:11:56.770544 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.770499    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-l56ns"]
Oct 29 22:11:56.865447 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.865421    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"]
Oct 29 22:11:56.882956 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:56.871573    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-pt68f"]
Oct 29 22:11:57.119650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.110033    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Oct 29 22:11:57.119650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.110091    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Oct 29 22:11:57.171930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.147201    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:11:57.171930 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.147253    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:11:57.453742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.452787    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:11:57.453742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.452853    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:11:57.588190 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.588163    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"
Oct 29 22:11:57.591072 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.591041    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp" event={"ID":"b25a95fc-3f00-488a-b750-1bb6383bcd95","Type":"ContainerDied","Data":"3268d64e7056c78da0c5c2ce899725266e6f5ff5079ac4d7a10271f74f184afb"}
Oct 29 22:11:57.591378 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.591362    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3268d64e7056c78da0c5c2ce899725266e6f5ff5079ac4d7a10271f74f184afb"
Oct 29 22:11:57.591497 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.591306    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"
Oct 29 22:11:57.766783 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.766752    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-openshift-ca\") pod \"b25a95fc-3f00-488a-b750-1bb6383bcd95\" (UID: \"b25a95fc-3f00-488a-b750-1bb6383bcd95\") "
Oct 29 22:11:57.768415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.767399    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-xks-ca\") pod \"b25a95fc-3f00-488a-b750-1bb6383bcd95\" (UID: \"b25a95fc-3f00-488a-b750-1bb6383bcd95\") "
Oct 29 22:11:57.768415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.767734    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25a95fc-3f00-488a-b750-1bb6383bcd95-kube-api-access-8ktgd\" (UniqueName: \"kubernetes.io/projected/b25a95fc-3f00-488a-b750-1bb6383bcd95-kube-api-access-8ktgd\") pod \"b25a95fc-3f00-488a-b750-1bb6383bcd95\" (UID: \"b25a95fc-3f00-488a-b750-1bb6383bcd95\") "
Oct 29 22:11:57.771745 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.771569    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-openshift-ca" pod "b25a95fc-3f00-488a-b750-1bb6383bcd95" (UID: "b25a95fc-3f00-488a-b750-1bb6383bcd95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:57.798646 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.793125    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" path="/var/lib/kubelet/pods/3379fba3-ce1e-4577-a370-ab3d675b4c4b/volumes"
Oct 29 22:11:57.798646 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.795000    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" path="/var/lib/kubelet/pods/58e47b26-da7b-4e93-aa45-16fc7cca65d0/volumes"
Oct 29 22:11:57.798646 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.795780    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" path="/var/lib/kubelet/pods/ddab3d72-73bf-4033-be01-6177dddff66a/volumes"
Oct 29 22:11:57.814059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.814008    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25a95fc-3f00-488a-b750-1bb6383bcd95-kube-api-access-8ktgd" pod "b25a95fc-3f00-488a-b750-1bb6383bcd95" (UID: "b25a95fc-3f00-488a-b750-1bb6383bcd95"). InnerVolumeSpecName "kube-api-access-8ktgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:57.843970 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.842163    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-xks-ca" pod "b25a95fc-3f00-488a-b750-1bb6383bcd95" (UID: "b25a95fc-3f00-488a-b750-1bb6383bcd95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:11:57.894665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.885792    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ktgd\" (UniqueName: \"kubernetes.io/projected/b25a95fc-3f00-488a-b750-1bb6383bcd95-kube-api-access-8ktgd\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:57.894665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.885900    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:57.894665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:57.885922    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b25a95fc-3f00-488a-b750-1bb6383bcd95-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:58.054531 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:58.052926    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:58.281647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:58.281512    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:58.446937 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:58.446815    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:11:58.636012 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:58.635457    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:11:58.636012 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:58.635535    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:11:58.842674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:58.839330    7082 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:11:58.842674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:58.839392    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:11:59.106274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.105942    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:59.106274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.106000    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:59.333238 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.332985    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:11:59.482809 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.482728    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-certs\") pod \"e9ac5793-4f48-4d6a-8390-19de6f99a814\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") "
Oct 29 22:11:59.483086 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.482865    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-podinfo\") pod \"e9ac5793-4f48-4d6a-8390-19de6f99a814\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") "
Oct 29 22:11:59.483086 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.482956    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-token\" (UniqueName: \"kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-token\") pod \"e9ac5793-4f48-4d6a-8390-19de6f99a814\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") "
Oct 29 22:11:59.483241 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.483125    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-certs" pod "e9ac5793-4f48-4d6a-8390-19de6f99a814" (UID: "e9ac5793-4f48-4d6a-8390-19de6f99a814"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:59.483241 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.483156    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ac5793-4f48-4d6a-8390-19de6f99a814-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ac5793-4f48-4d6a-8390-19de6f99a814-istiod-ca-cert\") pod \"e9ac5793-4f48-4d6a-8390-19de6f99a814\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") "
Oct 29 22:11:59.483241 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.483197    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-socket\") pod \"e9ac5793-4f48-4d6a-8390-19de6f99a814\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") "
Oct 29 22:11:59.483446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.483300    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-envoy\") pod \"e9ac5793-4f48-4d6a-8390-19de6f99a814\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") "
Oct 29 22:11:59.483446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.483370    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-data\") pod \"e9ac5793-4f48-4d6a-8390-19de6f99a814\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") "
Oct 29 22:11:59.483446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.483402    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-credential-socket\") pod \"e9ac5793-4f48-4d6a-8390-19de6f99a814\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") "
Oct 29 22:11:59.483601 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.483451    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-kube-api-access-9mrv9\" (UniqueName: \"kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-kube-api-access-9mrv9\") pod \"e9ac5793-4f48-4d6a-8390-19de6f99a814\" (UID: \"e9ac5793-4f48-4d6a-8390-19de6f99a814\") "
Oct 29 22:11:59.483601 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.483511    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ac5793-4f48-4d6a-8390-19de6f99a814-istiod-ca-cert" pod "e9ac5793-4f48-4d6a-8390-19de6f99a814" (UID: "e9ac5793-4f48-4d6a-8390-19de6f99a814"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:59.515653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.484601    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ac5793-4f48-4d6a-8390-19de6f99a814-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:59.515653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.504141    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-podinfo" pod "e9ac5793-4f48-4d6a-8390-19de6f99a814" (UID: "e9ac5793-4f48-4d6a-8390-19de6f99a814"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:59.515653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.511688    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-socket" pod "e9ac5793-4f48-4d6a-8390-19de6f99a814" (UID: "e9ac5793-4f48-4d6a-8390-19de6f99a814"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:59.516772 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.516713    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-data" pod "e9ac5793-4f48-4d6a-8390-19de6f99a814" (UID: "e9ac5793-4f48-4d6a-8390-19de6f99a814"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:59.517201 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.517165    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:59.518998 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.518957    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-credential-socket" pod "e9ac5793-4f48-4d6a-8390-19de6f99a814" (UID: "e9ac5793-4f48-4d6a-8390-19de6f99a814"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:59.520966 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.520925    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-envoy" pod "e9ac5793-4f48-4d6a-8390-19de6f99a814" (UID: "e9ac5793-4f48-4d6a-8390-19de6f99a814"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:59.521618 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.521575    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-kube-api-access-9mrv9" pod "e9ac5793-4f48-4d6a-8390-19de6f99a814" (UID: "e9ac5793-4f48-4d6a-8390-19de6f99a814"). InnerVolumeSpecName "kube-api-access-9mrv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:59.522128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.522098    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-token" pod "e9ac5793-4f48-4d6a-8390-19de6f99a814" (UID: "e9ac5793-4f48-4d6a-8390-19de6f99a814"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:59.635158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.634446    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:59.635158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.634494    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:59.635158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.634510    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mrv9\" (UniqueName: \"kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-kube-api-access-9mrv9\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:59.635158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.634523    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:59.635158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.634537    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:59.635158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.634551    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:59.635158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.634565    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ac5793-4f48-4d6a-8390-19de6f99a814-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:11:59.700657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.687829    7082 generic.go:356] "Generic (PLEG): container finished" podID="fca6221e-5b57-49a7-8a18-799752edb517" containerID="9e600ad115f340536d61e56ed2d7473645968cf243a1a9a3e895485c4874b8bf" exitCode=0
Oct 29 22:11:59.710671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.709791    7082 generic.go:356] "Generic (PLEG): container finished" podID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerID="dd2cb7245a324b8e286f9e9126733b31ef6ab17ef737554f47ade48e489de7c5" exitCode=0
Oct 29 22:11:59.713286 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.711318    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"
Oct 29 22:11:59.744491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.733728    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" event={"ID":"fca6221e-5b57-49a7-8a18-799752edb517","Type":"ContainerDied","Data":"9e600ad115f340536d61e56ed2d7473645968cf243a1a9a3e895485c4874b8bf"}
Oct 29 22:11:59.744491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.733837    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" event={"ID":"e9ac5793-4f48-4d6a-8390-19de6f99a814","Type":"ContainerDied","Data":"dd2cb7245a324b8e286f9e9126733b31ef6ab17ef737554f47ade48e489de7c5"}
Oct 29 22:11:59.744491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.733892    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf" event={"ID":"e9ac5793-4f48-4d6a-8390-19de6f99a814","Type":"ContainerDied","Data":"4b7ac3576c108eca9c6ce8c8ff368252ed2fca749f9694966f29272a6d9e112d"}
Oct 29 22:11:59.744491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.733920    7082 scope.go:122] "RemoveContainer" containerID="dd2cb7245a324b8e286f9e9126733b31ef6ab17ef737554f47ade48e489de7c5"
Oct 29 22:11:59.776654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.776614    7082 scope.go:122] "RemoveContainer" containerID="dd2cb7245a324b8e286f9e9126733b31ef6ab17ef737554f47ade48e489de7c5"
Oct 29 22:11:59.776970 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:11:59.776948    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd2cb7245a324b8e286f9e9126733b31ef6ab17ef737554f47ade48e489de7c5\": container with ID starting with dd2cb7245a324b8e286f9e9126733b31ef6ab17ef737554f47ade48e489de7c5 not found: ID does not exist" containerID="dd2cb7245a324b8e286f9e9126733b31ef6ab17ef737554f47ade48e489de7c5"
Oct 29 22:11:59.777058 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.776979    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd2cb7245a324b8e286f9e9126733b31ef6ab17ef737554f47ade48e489de7c5"} err="failed to get container status \"dd2cb7245a324b8e286f9e9126733b31ef6ab17ef737554f47ade48e489de7c5\": rpc error: code = NotFound desc = could not find container \"dd2cb7245a324b8e286f9e9126733b31ef6ab17ef737554f47ade48e489de7c5\": container with ID starting with dd2cb7245a324b8e286f9e9126733b31ef6ab17ef737554f47ade48e489de7c5 not found: ID does not exist"
Oct 29 22:11:59.831674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.827930    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"]
Oct 29 22:11:59.833058 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:11:59.832853    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-gbgtf"]
Oct 29 22:12:00.215900 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.215859    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 22:12:00.258688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.258030    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-envoy\") pod \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") "
Oct 29 22:12:00.258688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.258138    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-credential-socket\") pod \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") "
Oct 29 22:12:00.258688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.258182    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-podinfo\") pod \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") "
Oct 29 22:12:00.258688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.258311    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-token\") pod \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") "
Oct 29 22:12:00.258688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.258404    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-socket\") pod \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") "
Oct 29 22:12:00.258688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.258456    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istiod-ca-cert\") pod \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") "
Oct 29 22:12:00.258688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.258517    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-kube-api-access-kmk7m\" (UniqueName: \"kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-kube-api-access-kmk7m\") pod \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") "
Oct 29 22:12:00.258688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.258605    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-certs\") pod \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") "
Oct 29 22:12:00.259253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.258846    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-data\") pod \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\" (UID: \"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9\") "
Oct 29 22:12:00.260160 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.260062    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-data" pod "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" (UID: "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.325690 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.307904    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-kube-api-access-kmk7m" pod "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" (UID: "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9"). InnerVolumeSpecName "kube-api-access-kmk7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:00.328426 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.328364    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-envoy" pod "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" (UID: "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.328426 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.328422    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-socket" pod "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" (UID: "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.336978    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istiod-ca-cert" pod "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" (UID: "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.337098    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"]
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338290    7082 cpu_manager.go:414] "containerMap: removing container" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338312    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338341    7082 cpu_manager.go:414] "containerMap: removing container" podUID="b25a95fc-3f00-488a-b750-1bb6383bcd95" containerName="revoke-keys"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338348    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25a95fc-3f00-488a-b750-1bb6383bcd95" containerName="revoke-keys"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338365    7082 cpu_manager.go:414] "containerMap: removing container" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338373    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338390    7082 cpu_manager.go:414] "containerMap: removing container" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338399    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338408    7082 cpu_manager.go:414] "containerMap: removing container" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338415    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338440    7082 cpu_manager.go:414] "containerMap: removing container" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338447    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338469    7082 cpu_manager.go:414] "containerMap: removing container" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338476    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338487    7082 cpu_manager.go:414] "containerMap: removing container" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338495    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338977    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e47b26-da7b-4e93-aa45-16fc7cca65d0" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.338997    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ff3c7f-c721-44e3-b83e-b214b93fc2b8" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.339011    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.339025    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.339035    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.339052    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="3379fba3-ce1e-4577-a370-ab3d675b4c4b" containerName="istio-proxy"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.339066    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25a95fc-3f00-488a-b750-1bb6383bcd95" containerName="revoke-keys"
Oct 29 22:12:00.346652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.339078    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddab3d72-73bf-4033-be01-6177dddff66a" containerName="istio-proxy"
Oct 29 22:12:00.351020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.349789    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-certs" pod "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" (UID: "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.351020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.350985    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-credential-socket" pod "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" (UID: "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.358822 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.357966    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"]
Oct 29 22:12:00.361667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.360571    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.361667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.360891    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-podinfo" pod "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" (UID: "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:00.363491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.363361    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.363491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.363392    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.363491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.363407    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmk7m\" (UniqueName: \"kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-kube-api-access-kmk7m\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.363491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.363420    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.363491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.363433    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.363491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.363445    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.363491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.363457    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.363491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.363469    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.365995 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.365972    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-token" pod "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" (UID: "8c8cbca6-8215-4ea1-b10a-4e81fa8020d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:00.367386 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.367348    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"]
Oct 29 22:12:00.367751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.367695    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.391380 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.391300    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 22:12:00.439422 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.439374    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"]
Oct 29 22:12:00.444740 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.444710    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f82846-openshift-default-dockercfg-9s8bb\""
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.465907    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-envoy\") pod \"fca6221e-5b57-49a7-8a18-799752edb517\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") "
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.466011    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-credential-socket\") pod \"fca6221e-5b57-49a7-8a18-799752edb517\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") "
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.466053    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-socket\") pod \"fca6221e-5b57-49a7-8a18-799752edb517\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") "
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.466094    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-certs\") pod \"fca6221e-5b57-49a7-8a18-799752edb517\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") "
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.466198    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-kube-api-access-rbw57\" (UniqueName: \"kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-kube-api-access-rbw57\") pod \"fca6221e-5b57-49a7-8a18-799752edb517\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") "
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.466330    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-data\") pod \"fca6221e-5b57-49a7-8a18-799752edb517\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") "
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.466483    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fca6221e-5b57-49a7-8a18-799752edb517-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca6221e-5b57-49a7-8a18-799752edb517-istio-podinfo\") pod \"fca6221e-5b57-49a7-8a18-799752edb517\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") "
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.466545    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca6221e-5b57-49a7-8a18-799752edb517-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca6221e-5b57-49a7-8a18-799752edb517-istiod-ca-cert\") pod \"fca6221e-5b57-49a7-8a18-799752edb517\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") "
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.466615    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-istio-token\" (UniqueName: \"kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-istio-token\") pod \"fca6221e-5b57-49a7-8a18-799752edb517\" (UID: \"fca6221e-5b57-49a7-8a18-799752edb517\") "
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.467669    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.467731    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.467784    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.467824    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.467864    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.467902    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.467967    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.468002    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.468092    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.468348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.468148    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnl8c\" (UniqueName: \"kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-kube-api-access-mnl8c\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.479951 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.479918    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-data" pod "fca6221e-5b57-49a7-8a18-799752edb517" (UID: "fca6221e-5b57-49a7-8a18-799752edb517"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.480145 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.480006    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.480361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.480340    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.480526 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.480508    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.480670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.480651    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.480863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.480845    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.481030 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.481013    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.481216 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.481197    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.481353 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.481334    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-496b4\" (UniqueName: \"kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-kube-api-access-496b4\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.481441 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.480693    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-credential-socket" pod "fca6221e-5b57-49a7-8a18-799752edb517" (UID: "fca6221e-5b57-49a7-8a18-799752edb517"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.481524 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.481056    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca6221e-5b57-49a7-8a18-799752edb517-istiod-ca-cert" pod "fca6221e-5b57-49a7-8a18-799752edb517" (UID: "fca6221e-5b57-49a7-8a18-799752edb517"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:00.481602 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.481316    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-socket" pod "fca6221e-5b57-49a7-8a18-799752edb517" (UID: "fca6221e-5b57-49a7-8a18-799752edb517"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.482673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.481713    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.482673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.481335    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-certs" pod "fca6221e-5b57-49a7-8a18-799752edb517" (UID: "fca6221e-5b57-49a7-8a18-799752edb517"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.482673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.481789    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.512823 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.512760    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fca6221e-5b57-49a7-8a18-799752edb517-istio-podinfo" pod "fca6221e-5b57-49a7-8a18-799752edb517" (UID: "fca6221e-5b57-49a7-8a18-799752edb517"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:00.520565 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.519350    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-kube-api-access-rbw57" pod "fca6221e-5b57-49a7-8a18-799752edb517" (UID: "fca6221e-5b57-49a7-8a18-799752edb517"). InnerVolumeSpecName "kube-api-access-rbw57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:00.520565 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.520513    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-istio-token" pod "fca6221e-5b57-49a7-8a18-799752edb517" (UID: "fca6221e-5b57-49a7-8a18-799752edb517"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:00.521728 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.521700    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-envoy" pod "fca6221e-5b57-49a7-8a18-799752edb517" (UID: "fca6221e-5b57-49a7-8a18-799752edb517"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583063    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583116    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnl8c\" (UniqueName: \"kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-kube-api-access-mnl8c\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583154    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583187    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583214    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583228    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583269    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583302    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583341    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583356    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-496b4\" (UniqueName: \"kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-kube-api-access-496b4\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583504    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583521    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583548    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583568    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583586    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583605    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583680    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583698    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583806    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583818    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583829    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583837    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583847    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fca6221e-5b57-49a7-8a18-799752edb517-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583857    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbw57\" (UniqueName: \"kubernetes.io/projected/fca6221e-5b57-49a7-8a18-799752edb517-kube-api-access-rbw57\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583867    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fca6221e-5b57-49a7-8a18-799752edb517-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.583877    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fca6221e-5b57-49a7-8a18-799752edb517-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.584398    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.584854    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.585127    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.603084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.597370    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.606108 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.605002    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.606446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.606423    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.609793 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.609429    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.610790 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.610500    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.611175 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.611153    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.611798 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.611778    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.612368 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.612349    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.612685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.612605    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnl8c\" (UniqueName: \"kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-kube-api-access-mnl8c\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.612929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.612911    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.613860 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.613837    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.615969 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.615946    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.616313 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.616266    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.635539 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.635487    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.636814 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.636776    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-496b4\" (UniqueName: \"kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-kube-api-access-496b4\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.683290 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.683012    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:00.729529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.729492    7082 generic.go:356] "Generic (PLEG): container finished" podID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" containerID="7aad63497c30527e83474c40b35e6cbba76cac0a29d3eaa5bf4b095eee43dd27" exitCode=0
Oct 29 22:12:00.730236 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.729835    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"
Oct 29 22:12:00.731718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.729854    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" event={"ID":"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9","Type":"ContainerDied","Data":"7aad63497c30527e83474c40b35e6cbba76cac0a29d3eaa5bf4b095eee43dd27"}
Oct 29 22:12:00.731819 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.731760    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58" event={"ID":"8c8cbca6-8215-4ea1-b10a-4e81fa8020d9","Type":"ContainerDied","Data":"eadebf8bdd4a5943b18d0343d8ce2135ec5ed76eb2bde4a8ec4a46fafe6fb18b"}
Oct 29 22:12:00.731819 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.731792    7082 scope.go:122] "RemoveContainer" containerID="7aad63497c30527e83474c40b35e6cbba76cac0a29d3eaa5bf4b095eee43dd27"
Oct 29 22:12:00.734253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.734224    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp" event={"ID":"fca6221e-5b57-49a7-8a18-799752edb517","Type":"ContainerDied","Data":"34df19976a1970d58fba485cf77adf751824c62e6864d7eb11b005403715156d"}
Oct 29 22:12:00.734533 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.734513    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"
Oct 29 22:12:00.736605 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.735996    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:00.770086 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.769983    7082 scope.go:122] "RemoveContainer" containerID="7aad63497c30527e83474c40b35e6cbba76cac0a29d3eaa5bf4b095eee43dd27"
Oct 29 22:12:00.770662 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:12:00.770545    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aad63497c30527e83474c40b35e6cbba76cac0a29d3eaa5bf4b095eee43dd27\": container with ID starting with 7aad63497c30527e83474c40b35e6cbba76cac0a29d3eaa5bf4b095eee43dd27 not found: ID does not exist" containerID="7aad63497c30527e83474c40b35e6cbba76cac0a29d3eaa5bf4b095eee43dd27"
Oct 29 22:12:00.770662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.770605    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aad63497c30527e83474c40b35e6cbba76cac0a29d3eaa5bf4b095eee43dd27"} err="failed to get container status \"7aad63497c30527e83474c40b35e6cbba76cac0a29d3eaa5bf4b095eee43dd27\": rpc error: code = NotFound desc = could not find container \"7aad63497c30527e83474c40b35e6cbba76cac0a29d3eaa5bf4b095eee43dd27\": container with ID starting with 7aad63497c30527e83474c40b35e6cbba76cac0a29d3eaa5bf4b095eee43dd27 not found: ID does not exist"
Oct 29 22:12:00.771334 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.770664    7082 scope.go:122] "RemoveContainer" containerID="9e600ad115f340536d61e56ed2d7473645968cf243a1a9a3e895485c4874b8bf"
Oct 29 22:12:00.825659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.825049    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"]
Oct 29 22:12:00.832655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.829658    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-hbr58"]
Oct 29 22:12:00.954789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.954754    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"]
Oct 29 22:12:00.962396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:00.962364    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jh9hp"]
Oct 29 22:12:01.166716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.158855    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"]
Oct 29 22:12:01.171186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.171143    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:01.171310 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.171233    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:01.171310 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.171271    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:01.182248 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:12:01.182207    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd69823_b544_4f2b_90e8_08ed1a7c00d9.slice/crio-11a6f5dd7ad379eb25e1240024b9d383de13f9990d06929550abe0618653e547 WatchSource:0}: Error finding container 11a6f5dd7ad379eb25e1240024b9d383de13f9990d06929550abe0618653e547: Status 404 returned error can't find the container with id 11a6f5dd7ad379eb25e1240024b9d383de13f9990d06929550abe0618653e547
Oct 29 22:12:01.233408 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.233373    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"]
Oct 29 22:12:01.234213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.233908    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:01.234213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.233998    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:01.234213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.234066    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:01.634585 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.634552    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8cbca6-8215-4ea1-b10a-4e81fa8020d9" path="/var/lib/kubelet/pods/8c8cbca6-8215-4ea1-b10a-4e81fa8020d9/volumes"
Oct 29 22:12:01.635220 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.635198    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ac5793-4f48-4d6a-8390-19de6f99a814" path="/var/lib/kubelet/pods/e9ac5793-4f48-4d6a-8390-19de6f99a814/volumes"
Oct 29 22:12:01.635756 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.635737    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca6221e-5b57-49a7-8a18-799752edb517" path="/var/lib/kubelet/pods/fca6221e-5b57-49a7-8a18-799752edb517/volumes"
Oct 29 22:12:01.864382 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.864351    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp" event={"ID":"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10","Type":"ContainerStarted","Data":"1344d28f560ef40d94601ccbe58d0de0bc1823751a598b10a4802b0af034e1d0"}
Oct 29 22:12:01.864726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.864701    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp" event={"ID":"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10","Type":"ContainerStarted","Data":"57863ea8fb151b32ce4ad0c5d0f7c1bc683a8a360440c1c40570504801812db8"}
Oct 29 22:12:01.870429 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.869815    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc" event={"ID":"edd69823-b544-4f2b-90e8-08ed1a7c00d9","Type":"ContainerStarted","Data":"79a65f106c71393b77c61138cb0fbae9b5a7936ed2e116a4adca53791c194dd1"}
Oct 29 22:12:01.870429 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.869863    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc" event={"ID":"edd69823-b544-4f2b-90e8-08ed1a7c00d9","Type":"ContainerStarted","Data":"11a6f5dd7ad379eb25e1240024b9d383de13f9990d06929550abe0618653e547"}
Oct 29 22:12:01.935963 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.935916    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp" podStartSLOduration=1.93590263 podStartE2EDuration="1.93590263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:01.924584831 +0000 UTC m=+1494.424627131" watchObservedRunningTime="2026-10-29 22:12:01.93590263 +0000 UTC m=+1494.435944929"
Oct 29 22:12:01.980935 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:01.980876    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc" podStartSLOduration=1.9808574110000001 podStartE2EDuration="1.980857411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:01.972832802 +0000 UTC m=+1494.472875101" watchObservedRunningTime="2026-10-29 22:12:01.980857411 +0000 UTC m=+1494.480899711"
Oct 29 22:12:02.708660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:02.687994    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:02.746734 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:02.741868    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 29 22:12:02.746734 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:02.741932    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc" podUID="edd69823-b544-4f2b-90e8-08ed1a7c00d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 29 22:12:02.778835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:02.762764    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:02.778835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:02.763750    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 29 22:12:02.778835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:02.763804    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp" podUID="0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 29 22:12:03.688648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:03.687252    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 29 22:12:03.688648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:03.687310    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc" podUID="edd69823-b544-4f2b-90e8-08ed1a7c00d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 29 22:12:03.750647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:03.750205    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 29 22:12:03.750647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:03.750265    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp" podUID="0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 29 22:12:04.692192 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:04.691732    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 29 22:12:04.692192 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:04.691796    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc" podUID="edd69823-b544-4f2b-90e8-08ed1a7c00d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 29 22:12:04.753552 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:04.753519    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 29 22:12:04.754198 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:04.754156    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp" podUID="0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 29 22:12:05.684550 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:05.684514    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 29 22:12:05.684772 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:05.684571    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc" podUID="edd69823-b544-4f2b-90e8-08ed1a7c00d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 29 22:12:05.742652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:05.741453    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 29 22:12:05.742652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:05.741509    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp" podUID="0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 29 22:12:06.282716 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:12:06.282674    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8c570_3e0f_40b7_9696_81b84b10e8eb.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:12:06.685153 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:06.685118    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 29 22:12:06.685416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:06.685390    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc" podUID="edd69823-b544-4f2b-90e8-08ed1a7c00d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 29 22:12:06.739112 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:06.738983    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 29 22:12:06.739112 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:06.739058    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp" podUID="0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 29 22:12:07.708650 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:12:07.708007    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8c570_3e0f_40b7_9696_81b84b10e8eb.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:12:07.776904 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:07.776870    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Oct 29 22:12:07.780484 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:07.780305    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body=
Oct 29 22:12:07.780484 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:07.780444    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc" podUID="edd69823-b544-4f2b-90e8-08ed1a7c00d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused"
Oct 29 22:12:07.781124 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:07.780379    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp" podUID="0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Oct 29 22:12:08.741666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:08.736131    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:08.741666 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:08.739501    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:08.753603 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:08.753574    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:08.801080 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:08.754910    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:12:08.801080 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:08.755279    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:08.822190 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:08.822122    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:12:09.490397 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.490362    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"]
Oct 29 22:12:09.491570 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.491541    7082 generic.go:356] "Generic (PLEG): container finished" podID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerID="3d32fa4d317e07e36677770998a56d106e1161ca3a787fb645f99f769272193e" exitCode=0
Oct 29 22:12:09.491874 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.491853    7082 cpu_manager.go:414] "containerMap: removing container" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy"
Oct 29 22:12:09.491960 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.491876    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy"
Oct 29 22:12:09.492526 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.492495    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca6221e-5b57-49a7-8a18-799752edb517" containerName="istio-proxy"
Oct 29 22:12:09.498095 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.498023    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" event={"ID":"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6","Type":"ContainerDied","Data":"3d32fa4d317e07e36677770998a56d106e1161ca3a787fb645f99f769272193e"}
Oct 29 22:12:09.498260 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.498181    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.498859 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.498837    7082 scope.go:122] "RemoveContainer" containerID="3d32fa4d317e07e36677770998a56d106e1161ca3a787fb645f99f769272193e"
Oct 29 22:12:09.508891 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.508036    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:09.508891 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.508171    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:09.508891 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.508242    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:09.521910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.521876    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"]
Oct 29 22:12:09.527107 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.527004    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.528974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.527331    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.528974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.527474    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4dd\" (UniqueName: \"kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-kube-api-access-sg4dd\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.528974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.527985    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.528974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.528017    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.528974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.528128    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.528974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.528479    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.528974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.528588    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.530005 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.529459    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.633838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.633025    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.633838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.633106    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.633838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.633206    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4dd\" (UniqueName: \"kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-kube-api-access-sg4dd\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.633838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.633302    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.633838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.633333    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.633838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.633435    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.633838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.633476    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.633838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.633527    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.633838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.633612    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.634484 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.634093    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.635454 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.635425    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.636423 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.635927    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.636423 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.636088    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.636718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.636691    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.639058 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.638995    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.639617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.639577    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.661336 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.661305    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.680329 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.680073    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4dd\" (UniqueName: \"kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-kube-api-access-sg4dd\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:09.850457 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:09.850413    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:10.237987 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:10.237957    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"]
Oct 29 22:12:10.238406 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:12:10.238361    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3763e5_e3ad_4c90_b6c8_1581e3f45788.slice/crio-a432dba38394a461ce52a7b7422dca14351e30d8bf47ebcbeaadbb9774a4da86 WatchSource:0}: Error finding container a432dba38394a461ce52a7b7422dca14351e30d8bf47ebcbeaadbb9774a4da86: Status 404 returned error can't find the container with id a432dba38394a461ce52a7b7422dca14351e30d8bf47ebcbeaadbb9774a4da86
Oct 29 22:12:10.241888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:10.241854    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:10.241985 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:10.241927    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:10.241985 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:10.241966    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:10.540762 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:10.540608    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg" event={"ID":"9e3763e5-e3ad-4c90-b6c8-1581e3f45788","Type":"ContainerStarted","Data":"c046596fccd88a7f342eba164c1b75fc39643f1e38edcaefb35a68539680387a"}
Oct 29 22:12:10.540762 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:10.540736    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg" event={"ID":"9e3763e5-e3ad-4c90-b6c8-1581e3f45788","Type":"ContainerStarted","Data":"a432dba38394a461ce52a7b7422dca14351e30d8bf47ebcbeaadbb9774a4da86"}
Oct 29 22:12:10.548793 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:10.548373    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" event={"ID":"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6","Type":"ContainerStarted","Data":"7b4453e6564e1d172be133b09ec124a1627a6e6b6744dfa4a360197701c78b77"}
Oct 29 22:12:10.617611 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:10.617546    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg" podStartSLOduration=1.6175233279999999 podStartE2EDuration="1.617523328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:10.60335583 +0000 UTC m=+1503.103398139" watchObservedRunningTime="2026-10-29 22:12:10.617523328 +0000 UTC m=+1503.117565628"
Oct 29 22:12:10.853896 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:10.853807    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:11.052557 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:11.052514    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:12:11.053224 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:11.053195    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:12:11.053350 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:11.053247    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 22:12:11.852554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:11.852513    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 29 22:12:11.852875 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:11.852844    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg" podUID="9e3763e5-e3ad-4c90-b6c8-1581e3f45788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 29 22:12:12.053094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:12.053057    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:12:12.053675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:12.053119    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 22:12:12.851355 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:12.850868    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 29 22:12:12.851355 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:12.850926    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg" podUID="9e3763e5-e3ad-4c90-b6c8-1581e3f45788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 29 22:12:13.052326 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:13.052286    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:12:13.052868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:13.052834    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:12:13.052994 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:13.052893    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 22:12:13.851526 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:13.851489    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 29 22:12:13.852244 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:13.852199    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg" podUID="9e3763e5-e3ad-4c90-b6c8-1581e3f45788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 29 22:12:14.052295 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:14.052257    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:12:14.052534 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:14.052332    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 22:12:14.853428 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:14.852192    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 29 22:12:14.853428 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:14.852276    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg" podUID="9e3763e5-e3ad-4c90-b6c8-1581e3f45788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 29 22:12:15.052185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:15.051967    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:12:15.052185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:15.052044    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 22:12:15.457082 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:12:15.457036    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@21-32034-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf767f5a_7583_4f85_a50a_70c504a28898.slice/crio-a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf767f5a_7583_4f85_a50a_70c504a28898.slice/crio-conmon-a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:12:15.457320 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:12:15.456995    7082 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@21-32034-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf767f5a_7583_4f85_a50a_70c504a28898.slice/crio-a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf767f5a_7583_4f85_a50a_70c504a28898.slice/crio-conmon-a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:12:15.773155 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:15.773018    7082 generic.go:356] "Generic (PLEG): container finished" podID="bf767f5a-7583-4f85-a50a-70c504a28898" containerID="a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973" exitCode=0
Oct 29 22:12:15.773155 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:15.773087    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" event={"ID":"bf767f5a-7583-4f85-a50a-70c504a28898","Type":"ContainerDied","Data":"a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973"}
Oct 29 22:12:15.773864 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:15.773831    7082 scope.go:122] "RemoveContainer" containerID="a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973"
Oct 29 22:12:15.776067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:15.776020    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:15.776191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:15.776125    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:15.776191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:15.776176    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:15.851837 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:15.850994    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 29 22:12:15.851837 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:15.851064    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg" podUID="9e3763e5-e3ad-4c90-b6c8-1581e3f45788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 29 22:12:16.055670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:16.052319    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:12:16.055670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:16.052410    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 29 22:12:16.788514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:16.788475    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" event={"ID":"bf767f5a-7583-4f85-a50a-70c504a28898","Type":"ContainerStarted","Data":"d0d60ef12968e8110a371e797d6943db767c0e8451bea3dfbfa06998a851c6ee"}
Oct 29 22:12:16.852010 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:16.851970    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Oct 29 22:12:16.852328 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:16.852295    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg" podUID="9e3763e5-e3ad-4c90-b6c8-1581e3f45788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Oct 29 22:12:17.210759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:17.184284    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:12:17.245652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:17.237990    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:12:17.454191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:17.444709    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:12:17.454191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:17.444851    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:17.454191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:17.444893    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 29 22:12:17.954898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:17.954861    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:17.955681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:17.955658    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:17.959649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:17.957147    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:12:18.271714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:18.271640    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:12:18.401655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:18.400111    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:18.401655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:18.400172    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 29 22:12:19.402654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:19.399998    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:19.402654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:19.400067    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 29 22:12:20.393657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:20.388982    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:20.393657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:20.389044    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 29 22:12:21.407648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:21.407023    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:21.407648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:21.407088    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 29 22:12:22.408051 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:22.408002    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:22.416733 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:22.409123    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 29 22:12:23.386614 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:23.386576    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:23.387011 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:23.386977    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 29 22:12:24.390914 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:24.390880    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:12:24.392219 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:24.392195    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:12:27.342716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:27.342683    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"]
Oct 29 22:12:27.343218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:27.342967    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf" podUID="e98aa080-2663-42a6-b231-3b49fe819e26" containerName="payload-processing" containerID="cri-o://ab87208626bc016e35afea7d6b4e6763c06738177822d323501b0c665caed1ab" gracePeriod=30
Oct 29 22:12:27.377665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:27.377617    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"]
Oct 29 22:12:27.377920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:27.377885    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg" podUID="55bc3608-7cb5-4874-9ffa-7e62c0f8188b" containerName="payload-pre-processing" containerID="cri-o://10114397007db35351366bf6c9501f8f5c35a4e9fc358a2f9abc38b5e31ea927" gracePeriod=30
Oct 29 22:12:28.844057 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:28.843947    7082 generic.go:356] "Generic (PLEG): container finished" podID="e98aa080-2663-42a6-b231-3b49fe819e26" containerID="ab87208626bc016e35afea7d6b4e6763c06738177822d323501b0c665caed1ab" exitCode=0
Oct 29 22:12:28.844818 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:28.844755    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf" event={"ID":"e98aa080-2663-42a6-b231-3b49fe819e26","Type":"ContainerDied","Data":"ab87208626bc016e35afea7d6b4e6763c06738177822d323501b0c665caed1ab"}
Oct 29 22:12:28.900695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:28.900620    7082 generic.go:356] "Generic (PLEG): container finished" podID="55bc3608-7cb5-4874-9ffa-7e62c0f8188b" containerID="10114397007db35351366bf6c9501f8f5c35a4e9fc358a2f9abc38b5e31ea927" exitCode=0
Oct 29 22:12:28.901224 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:28.901161    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg" event={"ID":"55bc3608-7cb5-4874-9ffa-7e62c0f8188b","Type":"ContainerDied","Data":"10114397007db35351366bf6c9501f8f5c35a4e9fc358a2f9abc38b5e31ea927"}
Oct 29 22:12:28.924500 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:28.924434    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"]
Oct 29 22:12:28.929671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:28.926891    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"
Oct 29 22:12:29.009720 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.008018    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"
Oct 29 22:12:29.029658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.029065    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-edc2d3-mf8rp"]
Oct 29 22:12:29.036088 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.030792    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e98aa080-2663-42a6-b231-3b49fe819e26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98aa080-2663-42a6-b231-3b49fe819e26-plugins-config-volume\") pod \"e98aa080-2663-42a6-b231-3b49fe819e26\" (UID: \"e98aa080-2663-42a6-b231-3b49fe819e26\") "
Oct 29 22:12:29.036088 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.030844    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e98aa080-2663-42a6-b231-3b49fe819e26-kube-api-access-hlx9z\" (UniqueName: \"kubernetes.io/projected/e98aa080-2663-42a6-b231-3b49fe819e26-kube-api-access-hlx9z\") pod \"e98aa080-2663-42a6-b231-3b49fe819e26\" (UID: \"e98aa080-2663-42a6-b231-3b49fe819e26\") "
Oct 29 22:12:29.036088 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.034108    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e98aa080-2663-42a6-b231-3b49fe819e26-plugins-config-volume" pod "e98aa080-2663-42a6-b231-3b49fe819e26" (UID: "e98aa080-2663-42a6-b231-3b49fe819e26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:29.047688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.047653    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98aa080-2663-42a6-b231-3b49fe819e26-kube-api-access-hlx9z" pod "e98aa080-2663-42a6-b231-3b49fe819e26" (UID: "e98aa080-2663-42a6-b231-3b49fe819e26"). InnerVolumeSpecName "kube-api-access-hlx9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:29.055000 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:12:29.054951    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:12:29.136715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.133023    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-plugins-config-volume\") pod \"55bc3608-7cb5-4874-9ffa-7e62c0f8188b\" (UID: \"55bc3608-7cb5-4874-9ffa-7e62c0f8188b\") "
Oct 29 22:12:29.136715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.133476    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-kube-api-access-5qbvs\" (UniqueName: \"kubernetes.io/projected/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-kube-api-access-5qbvs\") pod \"55bc3608-7cb5-4874-9ffa-7e62c0f8188b\" (UID: \"55bc3608-7cb5-4874-9ffa-7e62c0f8188b\") "
Oct 29 22:12:29.136715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.134374    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e98aa080-2663-42a6-b231-3b49fe819e26-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:29.136715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.134398    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlx9z\" (UniqueName: \"kubernetes.io/projected/e98aa080-2663-42a6-b231-3b49fe819e26-kube-api-access-hlx9z\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:29.208655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.207284    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"]
Oct 29 22:12:29.226652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.210972    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-plugins-config-volume" pod "55bc3608-7cb5-4874-9ffa-7e62c0f8188b" (UID: "55bc3608-7cb5-4874-9ffa-7e62c0f8188b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:29.226652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.213611    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ae129-5czcn"]
Oct 29 22:12:29.248697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.236484    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:29.254178 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.254136    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-kube-api-access-5qbvs" pod "55bc3608-7cb5-4874-9ffa-7e62c0f8188b" (UID: "55bc3608-7cb5-4874-9ffa-7e62c0f8188b"). InnerVolumeSpecName "kube-api-access-5qbvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:29.338026 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.337990    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qbvs\" (UniqueName: \"kubernetes.io/projected/55bc3608-7cb5-4874-9ffa-7e62c0f8188b-kube-api-access-5qbvs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:29.638979 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.638941    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25a95fc-3f00-488a-b750-1bb6383bcd95" path="/var/lib/kubelet/pods/b25a95fc-3f00-488a-b750-1bb6383bcd95/volumes"
Oct 29 22:12:29.639582 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.639566    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9676fbf-6ef6-4ad1-a04c-5a962df40ac8" path="/var/lib/kubelet/pods/f9676fbf-6ef6-4ad1-a04c-5a962df40ac8/volumes"
Oct 29 22:12:29.913858 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.913826    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"
Oct 29 22:12:29.914416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.913835    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf" event={"ID":"e98aa080-2663-42a6-b231-3b49fe819e26","Type":"ContainerDied","Data":"60ae96ff627078de43c4641023474b035820d507b69f08219de1b3962267896a"}
Oct 29 22:12:29.914416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.914241    7082 scope.go:122] "RemoveContainer" containerID="ab87208626bc016e35afea7d6b4e6763c06738177822d323501b0c665caed1ab"
Oct 29 22:12:29.916825 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.916805    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"
Oct 29 22:12:29.917752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.917710    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg" event={"ID":"55bc3608-7cb5-4874-9ffa-7e62c0f8188b","Type":"ContainerDied","Data":"40e4f111e86e20afadf3f5024b6514e3166236fec0ef516ff8be1378e2eee461"}
Oct 29 22:12:29.929264 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.929246    7082 scope.go:122] "RemoveContainer" containerID="10114397007db35351366bf6c9501f8f5c35a4e9fc358a2f9abc38b5e31ea927"
Oct 29 22:12:29.968659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.968617    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"]
Oct 29 22:12:29.979687 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:29.979664    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-869f9df58d-g4hzf"]
Oct 29 22:12:30.006615 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.006587    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"]
Oct 29 22:12:30.013686 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.013660    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5d7fdf7bb7-cbrvg"]
Oct 29 22:12:30.413770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.413741    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"]
Oct 29 22:12:30.414528 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.414514    7082 cpu_manager.go:414] "containerMap: removing container" podUID="e98aa080-2663-42a6-b231-3b49fe819e26" containerName="payload-processing"
Oct 29 22:12:30.414575 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.414530    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e98aa080-2663-42a6-b231-3b49fe819e26" containerName="payload-processing"
Oct 29 22:12:30.414575 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.414569    7082 cpu_manager.go:414] "containerMap: removing container" podUID="55bc3608-7cb5-4874-9ffa-7e62c0f8188b" containerName="payload-pre-processing"
Oct 29 22:12:30.414575 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.414573    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55bc3608-7cb5-4874-9ffa-7e62c0f8188b" containerName="payload-pre-processing"
Oct 29 22:12:30.415211 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.414715    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="55bc3608-7cb5-4874-9ffa-7e62c0f8188b" containerName="payload-pre-processing"
Oct 29 22:12:30.415211 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.414731    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="e98aa080-2663-42a6-b231-3b49fe819e26" containerName="payload-processing"
Oct 29 22:12:30.418046 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.418029    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"
Oct 29 22:12:30.433075 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.433051    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:12:30.433478 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.433449    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:12:30.433616 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.433501    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:12:30.438675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.438653    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"]
Oct 29 22:12:30.444536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.444516    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:12:30.564459 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.564431    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb6hd\" (UniqueName: \"kubernetes.io/projected/e602c944-6a33-49cf-8ab5-1c225fbd4d44-kube-api-access-sb6hd\") pod \"maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl\" (UID: \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"
Oct 29 22:12:30.564723 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.564708    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl\" (UID: \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"
Oct 29 22:12:30.564793 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.564778    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl\" (UID: \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"
Oct 29 22:12:30.666973 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.666896    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb6hd\" (UniqueName: \"kubernetes.io/projected/e602c944-6a33-49cf-8ab5-1c225fbd4d44-kube-api-access-sb6hd\") pod \"maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl\" (UID: \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"
Oct 29 22:12:30.667140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.667122    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl\" (UID: \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"
Oct 29 22:12:30.667213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.667203    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl\" (UID: \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"
Oct 29 22:12:30.667846 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.667820    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl\" (UID: \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"
Oct 29 22:12:30.669944 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.669920    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl\" (UID: \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"
Oct 29 22:12:30.690315 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.690290    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb6hd\" (UniqueName: \"kubernetes.io/projected/e602c944-6a33-49cf-8ab5-1c225fbd4d44-kube-api-access-sb6hd\") pod \"maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl\" (UID: \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"
Oct 29 22:12:30.732361 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:30.732325    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"
Oct 29 22:12:31.061099 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:31.061072    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"]
Oct 29 22:12:31.062840 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:12:31.062816    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode602c944_6a33_49cf_8ab5_1c225fbd4d44.slice/crio-89db20a41b62e9ed9bca0665c76bc676a7dc7d018ceff053c714a97a6b65f0d7 WatchSource:0}: Error finding container 89db20a41b62e9ed9bca0665c76bc676a7dc7d018ceff053c714a97a6b65f0d7: Status 404 returned error can't find the container with id 89db20a41b62e9ed9bca0665c76bc676a7dc7d018ceff053c714a97a6b65f0d7
Oct 29 22:12:31.635509 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:31.635479    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55bc3608-7cb5-4874-9ffa-7e62c0f8188b" path="/var/lib/kubelet/pods/55bc3608-7cb5-4874-9ffa-7e62c0f8188b/volumes"
Oct 29 22:12:31.636102 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:31.636086    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e98aa080-2663-42a6-b231-3b49fe819e26" path="/var/lib/kubelet/pods/e98aa080-2663-42a6-b231-3b49fe819e26/volumes"
Oct 29 22:12:32.005391 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:32.005355    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl" event={"ID":"e602c944-6a33-49cf-8ab5-1c225fbd4d44","Type":"ContainerStarted","Data":"41bd3276d6f275b7b4333bd15fbee4e42e7f8c677fc322d3e9a2a205ca6aff38"}
Oct 29 22:12:32.005594 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:32.005421    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl" event={"ID":"e602c944-6a33-49cf-8ab5-1c225fbd4d44","Type":"ContainerStarted","Data":"89db20a41b62e9ed9bca0665c76bc676a7dc7d018ceff053c714a97a6b65f0d7"}
Oct 29 22:12:32.051328 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:32.051278    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl" podStartSLOduration=2.051264037 podStartE2EDuration="2.051264037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:32.046179285 +0000 UTC m=+1524.546221586" watchObservedRunningTime="2026-10-29 22:12:32.051264037 +0000 UTC m=+1524.551306337"
Oct 29 22:12:33.017881 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:33.017846    7082 generic.go:356] "Generic (PLEG): container finished" podID="e602c944-6a33-49cf-8ab5-1c225fbd4d44" containerID="41bd3276d6f275b7b4333bd15fbee4e42e7f8c677fc322d3e9a2a205ca6aff38" exitCode=0
Oct 29 22:12:33.018296 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:33.017938    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl" event={"ID":"e602c944-6a33-49cf-8ab5-1c225fbd4d44","Type":"ContainerDied","Data":"41bd3276d6f275b7b4333bd15fbee4e42e7f8c677fc322d3e9a2a205ca6aff38"}
Oct 29 22:12:34.351341 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:34.351312    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"
Oct 29 22:12:34.456577 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:34.456537    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-xks-ca\") pod \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\" (UID: \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\") "
Oct 29 22:12:34.456801 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:34.456671    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-openshift-ca\") pod \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\" (UID: \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\") "
Oct 29 22:12:34.456938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:34.456922    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e602c944-6a33-49cf-8ab5-1c225fbd4d44-kube-api-access-sb6hd\" (UniqueName: \"kubernetes.io/projected/e602c944-6a33-49cf-8ab5-1c225fbd4d44-kube-api-access-sb6hd\") pod \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\" (UID: \"e602c944-6a33-49cf-8ab5-1c225fbd4d44\") "
Oct 29 22:12:34.457115 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:34.457084    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-openshift-ca" pod "e602c944-6a33-49cf-8ab5-1c225fbd4d44" (UID: "e602c944-6a33-49cf-8ab5-1c225fbd4d44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:34.457773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:34.457749    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:34.459084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:34.459055    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-xks-ca" pod "e602c944-6a33-49cf-8ab5-1c225fbd4d44" (UID: "e602c944-6a33-49cf-8ab5-1c225fbd4d44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:12:34.459189 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:34.459132    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e602c944-6a33-49cf-8ab5-1c225fbd4d44-kube-api-access-sb6hd" pod "e602c944-6a33-49cf-8ab5-1c225fbd4d44" (UID: "e602c944-6a33-49cf-8ab5-1c225fbd4d44"). InnerVolumeSpecName "kube-api-access-sb6hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:34.559619 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:34.559590    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb6hd\" (UniqueName: \"kubernetes.io/projected/e602c944-6a33-49cf-8ab5-1c225fbd4d44-kube-api-access-sb6hd\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:34.559619 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:34.559617    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e602c944-6a33-49cf-8ab5-1c225fbd4d44-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:35.042842 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:35.042812    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl" event={"ID":"e602c944-6a33-49cf-8ab5-1c225fbd4d44","Type":"ContainerDied","Data":"89db20a41b62e9ed9bca0665c76bc676a7dc7d018ceff053c714a97a6b65f0d7"}
Oct 29 22:12:35.042842 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:35.042834    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"
Oct 29 22:12:35.043059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:35.042858    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89db20a41b62e9ed9bca0665c76bc676a7dc7d018ceff053c714a97a6b65f0d7"
Oct 29 22:12:35.922270 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:35.922232    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"]
Oct 29 22:12:35.922802 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:35.922712    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy" containerID="cri-o://bb0d3970ad0284d3e5003a858cdde4a4b38ad19021a5f1fad440b4c751cde795" gracePeriod=30
Oct 29 22:12:35.936574 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:35.936495    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"]
Oct 29 22:12:35.936822 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:35.936793    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" podUID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" containerName="istio-proxy" containerID="cri-o://f2d97a0488c0ff0bd2ec890e1c6a6df336bd08ee69dff90761c1164db9040044" gracePeriod=30
Oct 29 22:12:35.946066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:35.946038    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"]
Oct 29 22:12:35.946322 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:35.946296    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy" containerID="cri-o://4ef73f3dcfed5824f9c9e0a619a584045688182a941e877a323638046f068db5" gracePeriod=30
Oct 29 22:12:35.953102 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:35.953080    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"]
Oct 29 22:12:35.953363 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:35.953340    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" containerID="cri-o://fa8ea1c1e08a261bfc2dddb7a8ff9b454b86a1912b3386a4f42a2ac174cb2c49" gracePeriod=30
Oct 29 22:12:36.880418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:36.880381    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:12:36.880584 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:36.880439    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:12:37.924178 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:37.924143    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"]
Oct 29 22:12:37.924955 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:37.924939    7082 cpu_manager.go:414] "containerMap: removing container" podUID="e602c944-6a33-49cf-8ab5-1c225fbd4d44" containerName="revoke-keys"
Oct 29 22:12:37.924955 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:37.924956    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e602c944-6a33-49cf-8ab5-1c225fbd4d44" containerName="revoke-keys"
Oct 29 22:12:37.925097 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:37.925088    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="e602c944-6a33-49cf-8ab5-1c225fbd4d44" containerName="revoke-keys"
Oct 29 22:12:37.927814 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:37.927799    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"
Oct 29 22:12:37.955057 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:37.955008    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:12:37.955416 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:37.955391    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:12:37.963141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:37.963115    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"]
Oct 29 22:12:38.021188 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.021146    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:12:38.021376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.021350    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:12:38.028391 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.028364    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt\" (UID: \"b1461211-ff84-45be-b637-0e452114b0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"
Oct 29 22:12:38.028516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.028438    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt\" (UID: \"b1461211-ff84-45be-b637-0e452114b0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"
Oct 29 22:12:38.028611 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.028589    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts7dg\" (UniqueName: \"kubernetes.io/projected/b1461211-ff84-45be-b637-0e452114b0bf-kube-api-access-ts7dg\") pod \"maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt\" (UID: \"b1461211-ff84-45be-b637-0e452114b0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"
Oct 29 22:12:38.137754 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.131442    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt\" (UID: \"b1461211-ff84-45be-b637-0e452114b0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"
Oct 29 22:12:38.162703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.161798    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt\" (UID: \"b1461211-ff84-45be-b637-0e452114b0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"
Oct 29 22:12:38.162703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.161951    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts7dg\" (UniqueName: \"kubernetes.io/projected/b1461211-ff84-45be-b637-0e452114b0bf-kube-api-access-ts7dg\") pod \"maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt\" (UID: \"b1461211-ff84-45be-b637-0e452114b0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"
Oct 29 22:12:38.169730 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.153173    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt\" (UID: \"b1461211-ff84-45be-b637-0e452114b0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"
Oct 29 22:12:38.170321 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.170235    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt\" (UID: \"b1461211-ff84-45be-b637-0e452114b0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"
Oct 29 22:12:38.207112 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.207039    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts7dg\" (UniqueName: \"kubernetes.io/projected/b1461211-ff84-45be-b637-0e452114b0bf-kube-api-access-ts7dg\") pod \"maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt\" (UID: \"b1461211-ff84-45be-b637-0e452114b0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"
Oct 29 22:12:38.242756 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.242728    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"
Oct 29 22:12:38.273815 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.273780    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:12:38.273985 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.273846    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:12:38.559184 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:38.559155    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"]
Oct 29 22:12:38.559485 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:12:38.559458    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1461211_ff84_45be_b637_0e452114b0bf.slice/crio-97101bbc5cb53fdb96a2ac4187ce7d2095db4fb2d0e5bc036c6542f3856f525d WatchSource:0}: Error finding container 97101bbc5cb53fdb96a2ac4187ce7d2095db4fb2d0e5bc036c6542f3856f525d: Status 404 returned error can't find the container with id 97101bbc5cb53fdb96a2ac4187ce7d2095db4fb2d0e5bc036c6542f3856f525d
Oct 29 22:12:39.206692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.206604    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt" event={"ID":"b1461211-ff84-45be-b637-0e452114b0bf","Type":"ContainerStarted","Data":"e051064563523ece3e6f671a258438330084f5fd47afa3335916b5dec32e9294"}
Oct 29 22:12:39.207117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.207088    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt" event={"ID":"b1461211-ff84-45be-b637-0e452114b0bf","Type":"ContainerStarted","Data":"97101bbc5cb53fdb96a2ac4187ce7d2095db4fb2d0e5bc036c6542f3856f525d"}
Oct 29 22:12:39.245527 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.245476    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt" podStartSLOduration=2.24546028 podStartE2EDuration="2.24546028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:39.245217385 +0000 UTC m=+1531.745259685" watchObservedRunningTime="2026-10-29 22:12:39.24546028 +0000 UTC m=+1531.745502579"
Oct 29 22:12:39.592978 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.592946    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"]
Oct 29 22:12:39.597278 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.597257    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.643175 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.643144    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"]
Oct 29 22:12:39.689143 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.689114    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.689352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.689334    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.689530 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.689502    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.689595 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.689564    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.689660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.689593    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7650af16-12ae-49f4-93c0-9340911d29ab-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.689771 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.689753    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7650af16-12ae-49f4-93c0-9340911d29ab-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.689808 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.689798    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.689937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.689918    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfbt8\" (UniqueName: \"kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-kube-api-access-bfbt8\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.690027 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.690012    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.848704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.847285    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.848704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.847506    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.848704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.847734    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.848704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.847871    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.848704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.847912    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.848704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.847939    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7650af16-12ae-49f4-93c0-9340911d29ab-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.848704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.847984    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7650af16-12ae-49f4-93c0-9340911d29ab-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.848704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.848022    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.848704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.848123    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfbt8\" (UniqueName: \"kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-kube-api-access-bfbt8\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.859617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.858960    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.859617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.859260    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.877414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.873670    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.877414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.874011    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.877414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.874612    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.877414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.875244    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7650af16-12ae-49f4-93c0-9340911d29ab-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.880680 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.880594    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7650af16-12ae-49f4-93c0-9340911d29ab-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.932392 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.932362    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:39.933668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:39.933648    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfbt8\" (UniqueName: \"kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-kube-api-access-bfbt8\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:40.004780 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:40.004697    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:40.224884 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:40.224788    7082 generic.go:356] "Generic (PLEG): container finished" podID="b1461211-ff84-45be-b637-0e452114b0bf" containerID="e051064563523ece3e6f671a258438330084f5fd47afa3335916b5dec32e9294" exitCode=0
Oct 29 22:12:40.225349 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:40.224885    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt" event={"ID":"b1461211-ff84-45be-b637-0e452114b0bf","Type":"ContainerDied","Data":"e051064563523ece3e6f671a258438330084f5fd47afa3335916b5dec32e9294"}
Oct 29 22:12:40.356686 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:12:40.356653    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7650af16_12ae_49f4_93c0_9340911d29ab.slice/crio-dd63eb718ef23f1e22b24e923858d08a623b86b30943f674c28a31b2df7d5329 WatchSource:0}: Error finding container dd63eb718ef23f1e22b24e923858d08a623b86b30943f674c28a31b2df7d5329: Status 404 returned error can't find the container with id dd63eb718ef23f1e22b24e923858d08a623b86b30943f674c28a31b2df7d5329
Oct 29 22:12:40.358619 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:40.358593    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"]
Oct 29 22:12:40.360738 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:40.360685    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:40.361030 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:40.361001    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:40.361207 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:40.361188    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:12:40.877599 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:40.877566    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"]
Oct 29 22:12:40.908806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:40.908779    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"]
Oct 29 22:12:41.411670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.411509    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Oct 29 22:12:41.411670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.411565    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" podUID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Oct 29 22:12:41.451350 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.449255    7082 generic.go:356] "Generic (PLEG): container finished" podID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerID="fa8ea1c1e08a261bfc2dddb7a8ff9b454b86a1912b3386a4f42a2ac174cb2c49" exitCode=0
Oct 29 22:12:41.451350 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.449481    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" event={"ID":"c7b934d8-6393-4cfc-882e-07e965ed9d46","Type":"ContainerDied","Data":"fa8ea1c1e08a261bfc2dddb7a8ff9b454b86a1912b3386a4f42a2ac174cb2c49"}
Oct 29 22:12:41.451350 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.449546    7082 scope.go:122] "RemoveContainer" containerID="a97556ac71b2b236481c9226e1ba143d21954e41419390bd19a6ca73e9f7a087"
Oct 29 22:12:41.455955 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.455874    7082 generic.go:356] "Generic (PLEG): container finished" podID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" containerID="f2d97a0488c0ff0bd2ec890e1c6a6df336bd08ee69dff90761c1164db9040044" exitCode=0
Oct 29 22:12:41.456104 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.456026    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" event={"ID":"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5","Type":"ContainerDied","Data":"f2d97a0488c0ff0bd2ec890e1c6a6df336bd08ee69dff90761c1164db9040044"}
Oct 29 22:12:41.458474 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.458448    7082 generic.go:356] "Generic (PLEG): container finished" podID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerID="bb0d3970ad0284d3e5003a858cdde4a4b38ad19021a5f1fad440b4c751cde795" exitCode=0
Oct 29 22:12:41.458610 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.458552    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" event={"ID":"8de1a39a-2b26-4743-ab96-caa204152c4b","Type":"ContainerDied","Data":"bb0d3970ad0284d3e5003a858cdde4a4b38ad19021a5f1fad440b4c751cde795"}
Oct 29 22:12:41.459087 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.458747    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"]
Oct 29 22:12:41.463795 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.463772    7082 generic.go:356] "Generic (PLEG): container finished" podID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerID="4ef73f3dcfed5824f9c9e0a619a584045688182a941e877a323638046f068db5" exitCode=0
Oct 29 22:12:41.482746 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.482712    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h" event={"ID":"7650af16-12ae-49f4-93c0-9340911d29ab","Type":"ContainerStarted","Data":"1f675d5008e6cde64af9cde6a573a68a05494795bc074c35fd834cc4624fed70"}
Oct 29 22:12:41.482903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.482788    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"]
Oct 29 22:12:41.482903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.482822    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h" event={"ID":"7650af16-12ae-49f4-93c0-9340911d29ab","Type":"ContainerStarted","Data":"dd63eb718ef23f1e22b24e923858d08a623b86b30943f674c28a31b2df7d5329"}
Oct 29 22:12:41.482903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.482860    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" event={"ID":"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714","Type":"ContainerDied","Data":"4ef73f3dcfed5824f9c9e0a619a584045688182a941e877a323638046f068db5"}
Oct 29 22:12:41.483065 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.482963    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"
Oct 29 22:12:41.541649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.538320    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h" podStartSLOduration=2.538298805 podStartE2EDuration="2.538298805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:41.527956032 +0000 UTC m=+1534.027998357" watchObservedRunningTime="2026-10-29 22:12:41.538298805 +0000 UTC m=+1534.038341108"
Oct 29 22:12:41.636273 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.580090    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6\" (UID: \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"
Oct 29 22:12:41.636617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.636586    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6\" (UID: \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"
Oct 29 22:12:41.688469 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.688387    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzmnj\" (UniqueName: \"kubernetes.io/projected/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-kube-api-access-bzmnj\") pod \"maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6\" (UID: \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"
Oct 29 22:12:41.850357 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.829073    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6\" (UID: \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"
Oct 29 22:12:41.850357 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.829135    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6\" (UID: \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"
Oct 29 22:12:41.850357 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.829245    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzmnj\" (UniqueName: \"kubernetes.io/projected/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-kube-api-access-bzmnj\") pod \"maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6\" (UID: \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"
Oct 29 22:12:41.894686 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.889544    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6\" (UID: \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"
Oct 29 22:12:41.919714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:41.917754    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzmnj\" (UniqueName: \"kubernetes.io/projected/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-kube-api-access-bzmnj\") pod \"maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6\" (UID: \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"
Oct 29 22:12:42.076999 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.029017    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:42.172925 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.172892    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 29 22:12:42.173156 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.173132    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h" podUID="7650af16-12ae-49f4-93c0-9340911d29ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 29 22:12:42.174166 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.174143    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6\" (UID: \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"
Oct 29 22:12:42.217651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.214252    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"
Oct 29 22:12:42.261967 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.260198    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:12:42.274423 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.273362    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:12:42.355051 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.354968    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-credential-socket\") pod \"c7b934d8-6393-4cfc-882e-07e965ed9d46\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") "
Oct 29 22:12:42.355225 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.355055    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-envoy\") pod \"c7b934d8-6393-4cfc-882e-07e965ed9d46\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") "
Oct 29 22:12:42.355225 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.355173    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-podinfo\") pod \"c7b934d8-6393-4cfc-882e-07e965ed9d46\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") "
Oct 29 22:12:42.355225 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.355216    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-kube-api-access-twjs8\" (UniqueName: \"kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-kube-api-access-twjs8\") pod \"c7b934d8-6393-4cfc-882e-07e965ed9d46\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") "
Oct 29 22:12:42.355404 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.355331    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-data\") pod \"c7b934d8-6393-4cfc-882e-07e965ed9d46\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") "
Oct 29 22:12:42.355404 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.355363    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b934d8-6393-4cfc-882e-07e965ed9d46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b934d8-6393-4cfc-882e-07e965ed9d46-istiod-ca-cert\") pod \"c7b934d8-6393-4cfc-882e-07e965ed9d46\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") "
Oct 29 22:12:42.355404 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.355391    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-socket\") pod \"c7b934d8-6393-4cfc-882e-07e965ed9d46\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") "
Oct 29 22:12:42.355577 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.355419    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-token\" (UniqueName: \"kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-token\") pod \"c7b934d8-6393-4cfc-882e-07e965ed9d46\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") "
Oct 29 22:12:42.355686 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.355659    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-certs\") pod \"c7b934d8-6393-4cfc-882e-07e965ed9d46\" (UID: \"c7b934d8-6393-4cfc-882e-07e965ed9d46\") "
Oct 29 22:12:42.359091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.356771    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-certs" pod "c7b934d8-6393-4cfc-882e-07e965ed9d46" (UID: "c7b934d8-6393-4cfc-882e-07e965ed9d46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.359091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.356998    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-credential-socket" pod "c7b934d8-6393-4cfc-882e-07e965ed9d46" (UID: "c7b934d8-6393-4cfc-882e-07e965ed9d46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.359091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.357470    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-data" pod "c7b934d8-6393-4cfc-882e-07e965ed9d46" (UID: "c7b934d8-6393-4cfc-882e-07e965ed9d46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.359091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.358403    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-socket" pod "c7b934d8-6393-4cfc-882e-07e965ed9d46" (UID: "c7b934d8-6393-4cfc-882e-07e965ed9d46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.359091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.358571    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b934d8-6393-4cfc-882e-07e965ed9d46-istiod-ca-cert" pod "c7b934d8-6393-4cfc-882e-07e965ed9d46" (UID: "c7b934d8-6393-4cfc-882e-07e965ed9d46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:42.360612 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.359870    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-envoy" pod "c7b934d8-6393-4cfc-882e-07e965ed9d46" (UID: "c7b934d8-6393-4cfc-882e-07e965ed9d46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.364083 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.364048    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-kube-api-access-twjs8" pod "c7b934d8-6393-4cfc-882e-07e965ed9d46" (UID: "c7b934d8-6393-4cfc-882e-07e965ed9d46"). InnerVolumeSpecName "kube-api-access-twjs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.364863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.364840    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-podinfo" pod "c7b934d8-6393-4cfc-882e-07e965ed9d46" (UID: "c7b934d8-6393-4cfc-882e-07e965ed9d46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:42.366932 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.366906    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-token" pod "c7b934d8-6393-4cfc-882e-07e965ed9d46" (UID: "c7b934d8-6393-4cfc-882e-07e965ed9d46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.457390 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.457328    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8de1a39a-2b26-4743-ab96-caa204152c4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de1a39a-2b26-4743-ab96-caa204152c4b-istiod-ca-cert\") pod \"8de1a39a-2b26-4743-ab96-caa204152c4b\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") "
Oct 29 22:12:42.464186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.463268    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de1a39a-2b26-4743-ab96-caa204152c4b-istiod-ca-cert" pod "8de1a39a-2b26-4743-ab96-caa204152c4b" (UID: "8de1a39a-2b26-4743-ab96-caa204152c4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:42.464186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.463415    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-credential-socket\") pod \"8de1a39a-2b26-4743-ab96-caa204152c4b\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") "
Oct 29 22:12:42.464186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.463532    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-envoy\") pod \"8de1a39a-2b26-4743-ab96-caa204152c4b\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") "
Oct 29 22:12:42.464186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.463581    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-certs\") pod \"8de1a39a-2b26-4743-ab96-caa204152c4b\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") "
Oct 29 22:12:42.464186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.463720    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-data\") pod \"8de1a39a-2b26-4743-ab96-caa204152c4b\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") "
Oct 29 22:12:42.464186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.463820    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-kube-api-access-p4r78\" (UniqueName: \"kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-kube-api-access-p4r78\") pod \"8de1a39a-2b26-4743-ab96-caa204152c4b\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") "
Oct 29 22:12:42.464186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.463999    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-socket\") pod \"8de1a39a-2b26-4743-ab96-caa204152c4b\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") "
Oct 29 22:12:42.464186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.464061    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-podinfo\") pod \"8de1a39a-2b26-4743-ab96-caa204152c4b\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") "
Oct 29 22:12:42.466501 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.464965    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-token\" (UniqueName: \"kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-token\") pod \"8de1a39a-2b26-4743-ab96-caa204152c4b\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") "
Oct 29 22:12:42.466501 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.465998    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.466501 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.466022    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de1a39a-2b26-4743-ab96-caa204152c4b-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.466501 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.466037    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.466501 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.466052    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.466501 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.466065    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twjs8\" (UniqueName: \"kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-kube-api-access-twjs8\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.466501 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.466078    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.466501 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.466091    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b934d8-6393-4cfc-882e-07e965ed9d46-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.466501 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.466103    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.466501 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.466115    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b934d8-6393-4cfc-882e-07e965ed9d46-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.466501 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.466128    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b934d8-6393-4cfc-882e-07e965ed9d46-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.499091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.475020    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-socket" pod "8de1a39a-2b26-4743-ab96-caa204152c4b" (UID: "8de1a39a-2b26-4743-ab96-caa204152c4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.499091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.485125    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-data" pod "8de1a39a-2b26-4743-ab96-caa204152c4b" (UID: "8de1a39a-2b26-4743-ab96-caa204152c4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.499091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.485335    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-certs" pod "8de1a39a-2b26-4743-ab96-caa204152c4b" (UID: "8de1a39a-2b26-4743-ab96-caa204152c4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.499091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.485703    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-credential-socket" pod "8de1a39a-2b26-4743-ab96-caa204152c4b" (UID: "8de1a39a-2b26-4743-ab96-caa204152c4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.547151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.516822    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-kube-api-access-p4r78" pod "8de1a39a-2b26-4743-ab96-caa204152c4b" (UID: "8de1a39a-2b26-4743-ab96-caa204152c4b"). InnerVolumeSpecName "kube-api-access-p4r78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.565325 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.565240    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-envoy" pod "8de1a39a-2b26-4743-ab96-caa204152c4b" (UID: "8de1a39a-2b26-4743-ab96-caa204152c4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.568007 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.567952    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-podinfo" pod "8de1a39a-2b26-4743-ab96-caa204152c4b" (UID: "8de1a39a-2b26-4743-ab96-caa204152c4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:42.568307 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.568280    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-podinfo\") pod \"8de1a39a-2b26-4743-ab96-caa204152c4b\" (UID: \"8de1a39a-2b26-4743-ab96-caa204152c4b\") "
Oct 29 22:12:42.569259 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:12:42.569095    7082 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/8de1a39a-2b26-4743-ab96-caa204152c4b/volumes/kubernetes.io~downward-api/istio-podinfo
Oct 29 22:12:42.569259 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.569113    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-podinfo" pod "8de1a39a-2b26-4743-ab96-caa204152c4b" (UID: "8de1a39a-2b26-4743-ab96-caa204152c4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:42.570215 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.570193    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.570362 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.570348    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.570491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.570479    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.570912 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.570897    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.571657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.571641    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4r78\" (UniqueName: \"kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-kube-api-access-p4r78\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.573557 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.573540    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de1a39a-2b26-4743-ab96-caa204152c4b-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.573754 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.573738    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.573867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.572957    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"
Oct 29 22:12:42.574138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.574116    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:12:42.574229 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.571018    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-token" pod "8de1a39a-2b26-4743-ab96-caa204152c4b" (UID: "8de1a39a-2b26-4743-ab96-caa204152c4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.574229 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.572834    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z" event={"ID":"c7b934d8-6393-4cfc-882e-07e965ed9d46","Type":"ContainerDied","Data":"fbfd343edc056df11ed1e45dc7fba0ae3312e4bcca82c3b4851860cf6f1f0f87"}
Oct 29 22:12:42.574229 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.574205    7082 scope.go:122] "RemoveContainer" containerID="fa8ea1c1e08a261bfc2dddb7a8ff9b454b86a1912b3386a4f42a2ac174cb2c49"
Oct 29 22:12:42.574391 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.573174    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:12:42.575651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.575529    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"
Oct 29 22:12:42.575651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.575590    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4" event={"ID":"8de1a39a-2b26-4743-ab96-caa204152c4b","Type":"ContainerDied","Data":"281602ca0fabe632484f9cc6d66211cf1e5e99cfe67899d4d3eff2bbcaa14694"}
Oct 29 22:12:42.579743 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.579726    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"
Oct 29 22:12:42.579853 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.579825    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt" event={"ID":"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714","Type":"ContainerDied","Data":"37af08d97cc7e8b81af7b769895b73867f48176cee9c51143276e21826a7240c"}
Oct 29 22:12:42.659804 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.659772    7082 scope.go:122] "RemoveContainer" containerID="bb0d3970ad0284d3e5003a858cdde4a4b38ad19021a5f1fad440b4c751cde795"
Oct 29 22:12:42.674927 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.674899    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-podinfo\") pod \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") "
Oct 29 22:12:42.675059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.674978    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-envoy\") pod \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") "
Oct 29 22:12:42.676219 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.675852    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-token\" (UniqueName: \"kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-token\") pod \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") "
Oct 29 22:12:42.676363 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.676128    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"
Oct 29 22:12:42.676539 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.676496    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-kube-api-access-8m84t\" (UniqueName: \"kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-kube-api-access-8m84t\") pod \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") "
Oct 29 22:12:42.676909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.676892    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-socket\") pod \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") "
Oct 29 22:12:42.677185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.677168    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-credential-socket\") pod \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") "
Oct 29 22:12:42.677356 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.677166    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-socket" pod "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" (UID: "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.677440 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.677329    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-credential-socket" pod "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" (UID: "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.677539 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.677524    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istiod-ca-cert\") pod \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") "
Oct 29 22:12:42.677878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.677746    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istiod-ca-cert" pod "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" (UID: "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:42.678090 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.678036    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-certs" pod "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" (UID: "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.678238 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.678221    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-certs\") pod \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") "
Oct 29 22:12:42.678451 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.678434    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istiod-ca-cert\") pod \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") "
Oct 29 22:12:42.679255 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.679242    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-certs\") pod \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") "
Oct 29 22:12:42.679729 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.679716    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-data\") pod \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") "
Oct 29 22:12:42.680026 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.680013    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-socket\") pod \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") "
Oct 29 22:12:42.680304 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.680293    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-kube-api-access-jbzvg\" (UniqueName: \"kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-kube-api-access-jbzvg\") pod \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") "
Oct 29 22:12:42.680654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.680641    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-envoy\") pod \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") "
Oct 29 22:12:42.680975 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.680962    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-token\" (UniqueName: \"kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-token\") pod \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") "
Oct 29 22:12:42.681359 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.681347    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-credential-socket\") pod \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") "
Oct 29 22:12:42.681654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.681641    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-data\") pod \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\" (UID: \"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5\") "
Oct 29 22:12:42.681866 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.681850    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-podinfo\") pod \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\" (UID: \"9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714\") "
Oct 29 22:12:42.682920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.682903    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de1a39a-2b26-4743-ab96-caa204152c4b-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.684419 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.684410    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.684534 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.684523    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.684657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.684616    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.688753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.688736    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.688904 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.683959    7082 scope.go:122] "RemoveContainer" containerID="4ef73f3dcfed5824f9c9e0a619a584045688182a941e877a323638046f068db5"
Oct 29 22:12:42.689133 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.678805    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-envoy" pod "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" (UID: "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.764812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.678959    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istiod-ca-cert" pod "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" (UID: "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:42.765886 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.679045    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-podinfo" pod "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" (UID: "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:42.765886 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.679163    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-token" pod "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" (UID: "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.765886 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.679619    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-certs" pod "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" (UID: "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.766119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.679963    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-data" pod "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" (UID: "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.766119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.680249    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-socket" pod "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" (UID: "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.766119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.681414    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-kube-api-access-8m84t" pod "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" (UID: "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714"). InnerVolumeSpecName "kube-api-access-8m84t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.766119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.681774    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-credential-socket" pod "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" (UID: "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.766119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.681895    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-data" pod "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" (UID: "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.766119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.684249    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-envoy" pod "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" (UID: "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.766119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.684370    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-kube-api-access-jbzvg" pod "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" (UID: "c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5"). InnerVolumeSpecName "kube-api-access-jbzvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.766119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.762151    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-token" pod "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" (UID: "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.766119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.765773    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-podinfo" pod "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" (UID: "9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.794353    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1461211-ff84-45be-b637-0e452114b0bf-kube-api-access-ts7dg\" (UniqueName: \"kubernetes.io/projected/b1461211-ff84-45be-b637-0e452114b0bf-kube-api-access-ts7dg\") pod \"b1461211-ff84-45be-b637-0e452114b0bf\" (UID: \"b1461211-ff84-45be-b637-0e452114b0bf\") "
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.794538    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-openshift-ca\") pod \"b1461211-ff84-45be-b637-0e452114b0bf\" (UID: \"b1461211-ff84-45be-b637-0e452114b0bf\") "
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.794841    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-xks-ca\") pod \"b1461211-ff84-45be-b637-0e452114b0bf\" (UID: \"b1461211-ff84-45be-b637-0e452114b0bf\") "
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.795835    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.795860    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.795875    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.795891    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.795905    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.795919    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.795932    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.795944    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.795958    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m84t\" (UniqueName: \"kubernetes.io/projected/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-kube-api-access-8m84t\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.795971    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.795984    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.795997    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.796009    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.824681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.796023    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbzvg\" (UniqueName: \"kubernetes.io/projected/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5-kube-api-access-jbzvg\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.836571 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.836534    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-openshift-ca" pod "b1461211-ff84-45be-b637-0e452114b0bf" (UID: "b1461211-ff84-45be-b637-0e452114b0bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:42.840674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.840409    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-xks-ca" pod "b1461211-ff84-45be-b637-0e452114b0bf" (UID: "b1461211-ff84-45be-b637-0e452114b0bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:12:42.867401 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.867361    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1461211-ff84-45be-b637-0e452114b0bf-kube-api-access-ts7dg" pod "b1461211-ff84-45be-b637-0e452114b0bf" (UID: "b1461211-ff84-45be-b637-0e452114b0bf"). InnerVolumeSpecName "kube-api-access-ts7dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.874597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.874567    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"]
Oct 29 22:12:42.879041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.879017    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-hpgf4"]
Oct 29 22:12:42.969355 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.969325    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts7dg\" (UniqueName: \"kubernetes.io/projected/b1461211-ff84-45be-b637-0e452114b0bf-kube-api-access-ts7dg\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.969554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.969533    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.969748 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:42.969730    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1461211-ff84-45be-b637-0e452114b0bf-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:43.013745 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.008356    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 29 22:12:43.013745 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.008413    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h" podUID="7650af16-12ae-49f4-93c0-9340911d29ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 29 22:12:43.068279 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.064014    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"]
Oct 29 22:12:43.085651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.079497    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-5s69z"]
Oct 29 22:12:43.151899 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.151865    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"]
Oct 29 22:12:43.162335 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.162302    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"]
Oct 29 22:12:43.169388 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.169354    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-bfnbt"]
Oct 29 22:12:43.656716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.656692    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"
Oct 29 22:12:43.658105 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.658080    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"
Oct 29 22:12:43.665943 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.665919    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" path="/var/lib/kubelet/pods/8de1a39a-2b26-4743-ab96-caa204152c4b/volumes"
Oct 29 22:12:43.666896 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.666880    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" path="/var/lib/kubelet/pods/9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714/volumes"
Oct 29 22:12:43.667597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.667583    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" path="/var/lib/kubelet/pods/c7b934d8-6393-4cfc-882e-07e965ed9d46/volumes"
Oct 29 22:12:43.668455 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.668432    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs" event={"ID":"c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5","Type":"ContainerDied","Data":"4dc81ba12cbd7b2048e7cafff6f783ce0443ce6b9773534b87a229854bb76843"}
Oct 29 22:12:43.668547 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.668487    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt" event={"ID":"b1461211-ff84-45be-b637-0e452114b0bf","Type":"ContainerDied","Data":"97101bbc5cb53fdb96a2ac4187ce7d2095db4fb2d0e5bc036c6542f3856f525d"}
Oct 29 22:12:43.668547 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.668514    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97101bbc5cb53fdb96a2ac4187ce7d2095db4fb2d0e5bc036c6542f3856f525d"
Oct 29 22:12:43.668547 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.668526    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6" event={"ID":"a4095f5b-db7d-41c5-a147-0ac1b6d881a2","Type":"ContainerStarted","Data":"e1aafc5ff7ed681be0cc3c23949c4e3f3fb4d7f3ad6917f02c607b8ec9c617be"}
Oct 29 22:12:43.669110 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.668561    7082 scope.go:122] "RemoveContainer" containerID="f2d97a0488c0ff0bd2ec890e1c6a6df336bd08ee69dff90761c1164db9040044"
Oct 29 22:12:43.778354 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.778321    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"]
Oct 29 22:12:43.814652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:43.812386    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-vkfhs"]
Oct 29 22:12:44.044585 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:44.044201    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 29 22:12:44.044811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:44.044260    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h" podUID="7650af16-12ae-49f4-93c0-9340911d29ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 29 22:12:44.679849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:44.679816    7082 generic.go:356] "Generic (PLEG): container finished" podID="a4095f5b-db7d-41c5-a147-0ac1b6d881a2" containerID="89102eb4f3bee9e1ca6fea927b4a3f84e78b9fbb5c04b685a3dc5ae117067eaf" exitCode=0
Oct 29 22:12:44.680285 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:44.679920    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6" event={"ID":"a4095f5b-db7d-41c5-a147-0ac1b6d881a2","Type":"ContainerDied","Data":"89102eb4f3bee9e1ca6fea927b4a3f84e78b9fbb5c04b685a3dc5ae117067eaf"}
Oct 29 22:12:45.005197 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:45.005163    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 29 22:12:45.005471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:45.005444    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h" podUID="7650af16-12ae-49f4-93c0-9340911d29ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 29 22:12:45.858412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:45.858344    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" path="/var/lib/kubelet/pods/c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5/volumes"
Oct 29 22:12:45.945654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:45.924960    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s" podUID="7decf11f-4aa4-475a-8eea-e6bfb17fd48f" containerName="payload-pre-processing" containerID="cri-o://7a1b522104e3112c45ce6c539fbde2e591a9f93b88511b74d772323637847393" gracePeriod=25
Oct 29 22:12:46.019648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.017012    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 29 22:12:46.019648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.017069    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h" podUID="7650af16-12ae-49f4-93c0-9340911d29ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 29 22:12:46.465192 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.464865    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj" podUID="74cf46ea-ccc6-44e9-95c1-5d5aa2ded907" containerName="payload-processing" containerID="cri-o://b6c6ab8861412cfadababf6e115cb1960204cdb361997834c75b63ac2f993a65" gracePeriod=25
Oct 29 22:12:46.583744 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.583358    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"
Oct 29 22:12:46.683759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.682433    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-kube-api-access-bzmnj\" (UniqueName: \"kubernetes.io/projected/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-kube-api-access-bzmnj\") pod \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\" (UID: \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\") "
Oct 29 22:12:46.683759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.683090    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-openshift-ca\") pod \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\" (UID: \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\") "
Oct 29 22:12:46.683759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.683166    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-xks-ca\") pod \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\" (UID: \"a4095f5b-db7d-41c5-a147-0ac1b6d881a2\") "
Oct 29 22:12:46.763654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.740823    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-openshift-ca" pod "a4095f5b-db7d-41c5-a147-0ac1b6d881a2" (UID: "a4095f5b-db7d-41c5-a147-0ac1b6d881a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:46.763654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.746977    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-kube-api-access-bzmnj" pod "a4095f5b-db7d-41c5-a147-0ac1b6d881a2" (UID: "a4095f5b-db7d-41c5-a147-0ac1b6d881a2"). InnerVolumeSpecName "kube-api-access-bzmnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:46.770878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.770842    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-xks-ca" pod "a4095f5b-db7d-41c5-a147-0ac1b6d881a2" (UID: "a4095f5b-db7d-41c5-a147-0ac1b6d881a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:12:46.862473 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.862422    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:46.863130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.863110    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:46.863282 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.863266    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzmnj\" (UniqueName: \"kubernetes.io/projected/a4095f5b-db7d-41c5-a147-0ac1b6d881a2-kube-api-access-bzmnj\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:46.875941 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.875811    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"
Oct 29 22:12:46.900803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.884802    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6" event={"ID":"a4095f5b-db7d-41c5-a147-0ac1b6d881a2","Type":"ContainerDied","Data":"e1aafc5ff7ed681be0cc3c23949c4e3f3fb4d7f3ad6917f02c607b8ec9c617be"}
Oct 29 22:12:46.900803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.884882    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1aafc5ff7ed681be0cc3c23949c4e3f3fb4d7f3ad6917f02c607b8ec9c617be"
Oct 29 22:12:46.900803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.884950    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"
Oct 29 22:12:46.966258 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.966222    7082 generic.go:356] "Generic (PLEG): container finished" podID="74cf46ea-ccc6-44e9-95c1-5d5aa2ded907" containerID="b6c6ab8861412cfadababf6e115cb1960204cdb361997834c75b63ac2f993a65" exitCode=0
Oct 29 22:12:46.966452 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.966392    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj" event={"ID":"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907","Type":"ContainerDied","Data":"b6c6ab8861412cfadababf6e115cb1960204cdb361997834c75b63ac2f993a65"}
Oct 29 22:12:46.976165 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.976131    7082 generic.go:356] "Generic (PLEG): container finished" podID="7decf11f-4aa4-475a-8eea-e6bfb17fd48f" containerID="7a1b522104e3112c45ce6c539fbde2e591a9f93b88511b74d772323637847393" exitCode=0
Oct 29 22:12:46.976318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.976184    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s" event={"ID":"7decf11f-4aa4-475a-8eea-e6bfb17fd48f","Type":"ContainerDied","Data":"7a1b522104e3112c45ce6c539fbde2e591a9f93b88511b74d772323637847393"}
Oct 29 22:12:46.976318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.976224    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s" event={"ID":"7decf11f-4aa4-475a-8eea-e6bfb17fd48f","Type":"ContainerDied","Data":"222938c9d4fca66296d4bdb77b360deee9ccae399a4268663a439c9637a52faf"}
Oct 29 22:12:46.976318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.976250    7082 scope.go:122] "RemoveContainer" containerID="7a1b522104e3112c45ce6c539fbde2e591a9f93b88511b74d772323637847393"
Oct 29 22:12:46.976482 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:46.976429    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"
Oct 29 22:12:47.030896 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.021233    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 29 22:12:47.030896 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.021285    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h" podUID="7650af16-12ae-49f4-93c0-9340911d29ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 29 22:12:47.068216 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.068145    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-plugins-config-volume\") pod \"7decf11f-4aa4-475a-8eea-e6bfb17fd48f\" (UID: \"7decf11f-4aa4-475a-8eea-e6bfb17fd48f\") "
Oct 29 22:12:47.068400 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.068223    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-kube-api-access-q259q\" (UniqueName: \"kubernetes.io/projected/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-kube-api-access-q259q\") pod \"7decf11f-4aa4-475a-8eea-e6bfb17fd48f\" (UID: \"7decf11f-4aa4-475a-8eea-e6bfb17fd48f\") "
Oct 29 22:12:47.070327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.070299    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-plugins-config-volume" pod "7decf11f-4aa4-475a-8eea-e6bfb17fd48f" (UID: "7decf11f-4aa4-475a-8eea-e6bfb17fd48f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:47.071927 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.071893    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-kube-api-access-q259q" pod "7decf11f-4aa4-475a-8eea-e6bfb17fd48f" (UID: "7decf11f-4aa4-475a-8eea-e6bfb17fd48f"). InnerVolumeSpecName "kube-api-access-q259q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:47.078141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.078114    7082 scope.go:122] "RemoveContainer" containerID="7a1b522104e3112c45ce6c539fbde2e591a9f93b88511b74d772323637847393"
Oct 29 22:12:47.078564 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:12:47.078531    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1b522104e3112c45ce6c539fbde2e591a9f93b88511b74d772323637847393\": container with ID starting with 7a1b522104e3112c45ce6c539fbde2e591a9f93b88511b74d772323637847393 not found: ID does not exist" containerID="7a1b522104e3112c45ce6c539fbde2e591a9f93b88511b74d772323637847393"
Oct 29 22:12:47.079102 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.079071    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1b522104e3112c45ce6c539fbde2e591a9f93b88511b74d772323637847393"} err="failed to get container status \"7a1b522104e3112c45ce6c539fbde2e591a9f93b88511b74d772323637847393\": rpc error: code = NotFound desc = could not find container \"7a1b522104e3112c45ce6c539fbde2e591a9f93b88511b74d772323637847393\": container with ID starting with 7a1b522104e3112c45ce6c539fbde2e591a9f93b88511b74d772323637847393 not found: ID does not exist"
Oct 29 22:12:47.175860 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.175183    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:47.176201 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.176175    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q259q\" (UniqueName: \"kubernetes.io/projected/7decf11f-4aa4-475a-8eea-e6bfb17fd48f-kube-api-access-q259q\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:47.261591 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.261558    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"
Oct 29 22:12:47.277178 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.277151    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-plugins-config-volume\") pod \"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907\" (UID: \"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907\") "
Oct 29 22:12:47.277320 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.277265    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-kube-api-access-m8mzd\" (UniqueName: \"kubernetes.io/projected/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-kube-api-access-m8mzd\") pod \"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907\" (UID: \"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907\") "
Oct 29 22:12:47.278214 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.278186    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-plugins-config-volume" pod "74cf46ea-ccc6-44e9-95c1-5d5aa2ded907" (UID: "74cf46ea-ccc6-44e9-95c1-5d5aa2ded907"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:47.279995 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.279975    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-kube-api-access-m8mzd" pod "74cf46ea-ccc6-44e9-95c1-5d5aa2ded907" (UID: "74cf46ea-ccc6-44e9-95c1-5d5aa2ded907"). InnerVolumeSpecName "kube-api-access-m8mzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:47.379082 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.379054    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8mzd\" (UniqueName: \"kubernetes.io/projected/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-kube-api-access-m8mzd\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:47.379082 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.379080    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:47.449719 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.449682    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"]
Oct 29 22:12:47.451024 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.450996    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-27cda7-7c46895b59-pdq5s"]
Oct 29 22:12:47.643821 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:47.643749    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7decf11f-4aa4-475a-8eea-e6bfb17fd48f" path="/var/lib/kubelet/pods/7decf11f-4aa4-475a-8eea-e6bfb17fd48f/volumes"
Oct 29 22:12:48.020683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:48.020553    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj" event={"ID":"74cf46ea-ccc6-44e9-95c1-5d5aa2ded907","Type":"ContainerDied","Data":"6e4862ac2389306693f2fc4ef13bf4a067343d15b72d5c025f387df55473dc41"}
Oct 29 22:12:48.021255 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:48.021222    7082 scope.go:122] "RemoveContainer" containerID="b6c6ab8861412cfadababf6e115cb1960204cdb361997834c75b63ac2f993a65"
Oct 29 22:12:48.021255 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:48.020612    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"
Oct 29 22:12:48.027203 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:48.027182    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:48.027499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:48.027480    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:48.028483 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:48.028465    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:12:48.137849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:48.137766    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"]
Oct 29 22:12:48.146673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:48.144025    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-27cda7-5c5854cd64-v8zvj"]
Oct 29 22:12:48.371846 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:48.371814    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"]
Oct 29 22:12:49.672650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:49.671743    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74cf46ea-ccc6-44e9-95c1-5d5aa2ded907" path="/var/lib/kubelet/pods/74cf46ea-ccc6-44e9-95c1-5d5aa2ded907/volumes"
Oct 29 22:12:51.353686 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:51.352915    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:12:53.470653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:53.450795    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b" podUID="7931685e-46cf-4f26-9af5-d0e43e44997d" containerName="payload-processing" containerID="cri-o://25401049a108a54217a886fd6522915403252c84d886bf5736a4501a115e9a25" gracePeriod=25
Oct 29 22:12:54.059646 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:54.058549    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:54.078114 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:54.077235    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:54.823656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:54.821471    7082 generic.go:356] "Generic (PLEG): container finished" podID="7931685e-46cf-4f26-9af5-d0e43e44997d" containerID="25401049a108a54217a886fd6522915403252c84d886bf5736a4501a115e9a25" exitCode=0
Oct 29 22:12:54.823656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:54.821759    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b" event={"ID":"7931685e-46cf-4f26-9af5-d0e43e44997d","Type":"ContainerDied","Data":"25401049a108a54217a886fd6522915403252c84d886bf5736a4501a115e9a25"}
Oct 29 22:12:55.146701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:55.143830    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"
Oct 29 22:12:55.292874 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:55.292834    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7931685e-46cf-4f26-9af5-d0e43e44997d-kube-api-access-xvkdc\" (UniqueName: \"kubernetes.io/projected/7931685e-46cf-4f26-9af5-d0e43e44997d-kube-api-access-xvkdc\") pod \"7931685e-46cf-4f26-9af5-d0e43e44997d\" (UID: \"7931685e-46cf-4f26-9af5-d0e43e44997d\") "
Oct 29 22:12:55.293083 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:55.292960    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7931685e-46cf-4f26-9af5-d0e43e44997d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7931685e-46cf-4f26-9af5-d0e43e44997d-plugins-config-volume\") pod \"7931685e-46cf-4f26-9af5-d0e43e44997d\" (UID: \"7931685e-46cf-4f26-9af5-d0e43e44997d\") "
Oct 29 22:12:55.332915 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:55.332858    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7931685e-46cf-4f26-9af5-d0e43e44997d-kube-api-access-xvkdc" pod "7931685e-46cf-4f26-9af5-d0e43e44997d" (UID: "7931685e-46cf-4f26-9af5-d0e43e44997d"). InnerVolumeSpecName "kube-api-access-xvkdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:55.334300 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:55.334268    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7931685e-46cf-4f26-9af5-d0e43e44997d-plugins-config-volume" pod "7931685e-46cf-4f26-9af5-d0e43e44997d" (UID: "7931685e-46cf-4f26-9af5-d0e43e44997d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:55.424705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:55.409336    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvkdc\" (UniqueName: \"kubernetes.io/projected/7931685e-46cf-4f26-9af5-d0e43e44997d-kube-api-access-xvkdc\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:55.424705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:55.409373    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7931685e-46cf-4f26-9af5-d0e43e44997d-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:12:56.006654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:55.988840    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"
Oct 29 22:12:56.006654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:55.988909    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b" event={"ID":"7931685e-46cf-4f26-9af5-d0e43e44997d","Type":"ContainerDied","Data":"744b97b1c05d380c819dd69d5c703c6e476eef6a9c3e594ce7b6cabb48d1d14f"}
Oct 29 22:12:56.006654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:55.989010    7082 scope.go:122] "RemoveContainer" containerID="25401049a108a54217a886fd6522915403252c84d886bf5736a4501a115e9a25"
Oct 29 22:12:56.072745 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:56.071779    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:12:56.077175 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:56.077140    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:56.125651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:56.119730    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:56.184759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:56.184673    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"]
Oct 29 22:12:56.223250 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:56.222721    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:56.223496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:56.223478    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-48c52b-6b84fd4d5b-6lx5b"]
Oct 29 22:12:56.285747 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:56.285704    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.096744 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:57.092179    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.117052 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:57.114556    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:12:57.125098 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:57.120609    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.152499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:57.140904    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:12:57.152499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:57.141831    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:12:57.152499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:57.142324    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:12:57.303427 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:57.303375    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.304385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:57.303763    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:12:57.318780 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:57.316157    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:12:57.318780 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:57.316229    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:12:57.844684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:57.844607    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:58.081183 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:58.081144    7082 scope.go:122] "RemoveContainer" containerID="080f0671dc710ddbdd6e8e9b25565bf913b6495eb50ed049fc7573a5faaae708"
Oct 29 22:12:58.119720 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:58.106148    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:58.137686 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:58.124246    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:58.178659 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:58.142015    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:12:58.527712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:58.501900    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:58.815276 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:58.815221    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:12:59.088707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:58.985268    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7931685e-46cf-4f26-9af5-d0e43e44997d" path="/var/lib/kubelet/pods/7931685e-46cf-4f26-9af5-d0e43e44997d/volumes"
Oct 29 22:12:59.088707 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:12:59.030984    7082 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.399s"
Oct 29 22:12:59.361700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:59.347314    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:12:59.424505 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:59.389802    7082 scope.go:122] "RemoveContainer" containerID="0e480829ecdefd9db1337b0295e79353b729e312eb876e770eb77475955702cb"
Oct 29 22:12:59.511651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:59.459128    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:12:59.662707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:59.603331    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:12:59.929274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:12:59.929165    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tbk7p" podUID="0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:13:00.015797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:00.015758    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:00.016105 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:00.016070    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.071732 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:00.048798    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.080677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:00.072094    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.128012 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:00.127956    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:00.128356 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:00.128318    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.213709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:00.209657    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:00.213709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:00.209766    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.239358 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:13:00.239304    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:13:00.326652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:00.322797    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:00.326652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:00.322879    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.467705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:00.464151    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.483907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:00.466899    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:13:00.675942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:00.675899    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.069366 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:01.069327    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:13:01.078841 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:01.078800    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded"
Oct 29 22:13:01.127797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:01.127747    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.147410 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:01.147370    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:01.147722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:01.147688    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.150472 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:01.150449    7082 scope.go:122] "RemoveContainer" containerID="4b70a76d3517e661c134b03cc4ced542cd673f1c2d851d70573bc9ca511d3971"
Oct 29 22:13:01.498662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:01.483987    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.498662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:01.485039    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.948477 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:01.903755    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.017138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:02.016686    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:02.017138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:02.016756    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.095685 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:02.094889    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.128835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:02.128755    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:13:02.192749 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:02.176711    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded"
Oct 29 22:13:02.490774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:02.490730    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.491388 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:02.491357    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:13:02.905972 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:02.904945    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:13:02.996919 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:02.996852    7082 patch_prober.go:29] interesting pod/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:02.997221 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:02.997171    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.126691 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:03.105765    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:13:03.149683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:03.129861    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.504162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:03.504119    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:13:03.523897 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:03.504410    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:13:03.600871 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:03.545343    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.686966 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:03.686934    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:03.687167 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:03.687139    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.944800 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:03.938898    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.114186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:04.114127    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.143460 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:04.143400    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.457663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:04.456936    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:04.457663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:04.457001    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.544655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:04.538011    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:13:04.616517 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:04.616471    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:13:04.744839 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:04.744739    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:04.745092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:04.745066    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.782651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:04.761728    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.994655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:04.982355    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:05.116773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:05.116720    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:05.139657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:05.139167    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:13:05.560710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:05.556248    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:13:05.733200 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:05.699670    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:05.983129 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:05.983029    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:13:06.259545 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:06.255739    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:06.552677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:06.552645    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:13:06.552869 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:06.552729    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:13:06.552869 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:06.552784    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:13:06.552869 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:06.552827    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:13:07.539672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:07.539552    7082 generic.go:356] "Generic (PLEG): container finished" podID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerID="5e5971e8c0ee066cbbaa007def83ced34ee096d0d8d2375740f7302916d71340" exitCode=0
Oct 29 22:13:07.539672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:07.539660    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" event={"ID":"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248","Type":"ContainerDied","Data":"5e5971e8c0ee066cbbaa007def83ced34ee096d0d8d2375740f7302916d71340"}
Oct 29 22:13:07.540130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:07.540058    7082 scope.go:122] "RemoveContainer" containerID="5e5971e8c0ee066cbbaa007def83ced34ee096d0d8d2375740f7302916d71340"
Oct 29 22:13:07.541560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:07.541530    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:13:07.541622 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:07.541608    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:13:07.541712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:07.541687    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:13:08.557925 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:08.557887    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" event={"ID":"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248","Type":"ContainerStarted","Data":"1172f637051c7d5c6e26b478bea97d1a88e00b7ffc01afe8fa99a9fec39d0f1d"}
Oct 29 22:13:09.536438 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:09.536402    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:13:09.537507 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:09.537467    7082 patch_prober.go:29] interesting pod/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 29 22:13:09.537670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:09.537536    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 29 22:13:10.055822 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.055789    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"]
Oct 29 22:13:10.056600 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056583    7082 cpu_manager.go:414] "containerMap: removing container" podUID="b1461211-ff84-45be-b637-0e452114b0bf" containerName="revoke-keys"
Oct 29 22:13:10.056703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056602    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1461211-ff84-45be-b637-0e452114b0bf" containerName="revoke-keys"
Oct 29 22:13:10.056703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056616    7082 cpu_manager.go:414] "containerMap: removing container" podUID="a4095f5b-db7d-41c5-a147-0ac1b6d881a2" containerName="revoke-keys"
Oct 29 22:13:10.056703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056621    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4095f5b-db7d-41c5-a147-0ac1b6d881a2" containerName="revoke-keys"
Oct 29 22:13:10.056703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056645    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy"
Oct 29 22:13:10.056703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056651    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy"
Oct 29 22:13:10.056703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056662    7082 cpu_manager.go:414] "containerMap: removing container" podUID="74cf46ea-ccc6-44e9-95c1-5d5aa2ded907" containerName="payload-processing"
Oct 29 22:13:10.056703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056667    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74cf46ea-ccc6-44e9-95c1-5d5aa2ded907" containerName="payload-processing"
Oct 29 22:13:10.056703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056676    7082 cpu_manager.go:414] "containerMap: removing container" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy"
Oct 29 22:13:10.056703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056681    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy"
Oct 29 22:13:10.056703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056687    7082 cpu_manager.go:414] "containerMap: removing container" podUID="7decf11f-4aa4-475a-8eea-e6bfb17fd48f" containerName="payload-pre-processing"
Oct 29 22:13:10.056703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056692    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7decf11f-4aa4-475a-8eea-e6bfb17fd48f" containerName="payload-pre-processing"
Oct 29 22:13:10.056703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056705    7082 cpu_manager.go:414] "containerMap: removing container" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy"
Oct 29 22:13:10.056703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056710    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056722    7082 cpu_manager.go:414] "containerMap: removing container" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056727    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056743    7082 cpu_manager.go:414] "containerMap: removing container" podUID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" containerName="istio-proxy"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056749    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" containerName="istio-proxy"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056756    7082 cpu_manager.go:414] "containerMap: removing container" podUID="7931685e-46cf-4f26-9af5-d0e43e44997d" containerName="payload-processing"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056761    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7931685e-46cf-4f26-9af5-d0e43e44997d" containerName="payload-processing"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056864    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="74cf46ea-ccc6-44e9-95c1-5d5aa2ded907" containerName="payload-processing"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056871    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056877    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="7decf11f-4aa4-475a-8eea-e6bfb17fd48f" containerName="payload-pre-processing"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056885    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de1a39a-2b26-4743-ab96-caa204152c4b" containerName="istio-proxy"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056895    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c0f2c8-2ada-4167-9f3e-f9095ce2ecf5" containerName="istio-proxy"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056901    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="7931685e-46cf-4f26-9af5-d0e43e44997d" containerName="payload-processing"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056913    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1461211-ff84-45be-b637-0e452114b0bf" containerName="revoke-keys"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056918    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4095f5b-db7d-41c5-a147-0ac1b6d881a2" containerName="revoke-keys"
Oct 29 22:13:10.057130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.056924    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb3aa0d-adb0-4e8f-8a8b-1e184e0be714" containerName="istio-proxy"
Oct 29 22:13:10.059590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.059575    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.117302 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.117264    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-75db5b-openshift-default-dockercfg-mwm8c\""
Oct 29 22:13:10.124434 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.124400    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"]
Oct 29 22:13:10.130895 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.130867    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"]
Oct 29 22:13:10.132029 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.131996    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b934d8-6393-4cfc-882e-07e965ed9d46" containerName="istio-proxy"
Oct 29 22:13:10.191662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.184986    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.191662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.185149    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.191662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.185179    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.191662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.185254    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.191662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.185354    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs47w\" (UniqueName: \"kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-kube-api-access-gs47w\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.191662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.185428    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.191662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.185457    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.191662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.185670    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.191662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.185734    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.362724    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.362762    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.362819    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.362882    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs47w\" (UniqueName: \"kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-kube-api-access-gs47w\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.362927    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.362945    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.363053    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.363089    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.363121    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.363385    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"]
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.363432    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.363531    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.363601    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.364108    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.364241    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.366339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.364854    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.367736 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.366350    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.369200 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.369154    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.423651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.423214    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.423651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.423566    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs47w\" (UniqueName: \"kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-kube-api-access-gs47w\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.464356 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.464328    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.464476 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.464373    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.464476 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.464439    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.464554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.464505    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.464554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.464529    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.464727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.464698    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.464830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.464813    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.465010 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.464988    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.465066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.465028    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kk29\" (UniqueName: \"kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-kube-api-access-4kk29\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.536334 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.536291    7082 patch_prober.go:29] interesting pod/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 29 22:13:10.536540 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.536349    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 29 22:13:10.567060 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.567028    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.567714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.567179    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.567714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.567204    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kk29\" (UniqueName: \"kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-kube-api-access-4kk29\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.567714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.567304    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.567714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.567336    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.567714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.567358    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.567714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.567428    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.567714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.567447    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.567714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.567511    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.568162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.567715    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.568162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.567805    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.568162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.568136    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.568268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.568215    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.568448 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.568427    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.569750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.569726    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.570429 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.570407    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.617445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.617315    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kk29\" (UniqueName: \"kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-kube-api-access-4kk29\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.620023 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.619998    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:10.674186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.674147    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:10.759360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:10.759149    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:11.051240 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.051210    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"]
Oct 29 22:13:11.051712 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:13:11.051686    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027d1e1a_2888_45b7_b47e_2b62ca191f7a.slice/crio-ab6455286795e0e25599c7b0704c02d5ca5eaeac8906fe31e84547ecc3f224e2 WatchSource:0}: Error finding container ab6455286795e0e25599c7b0704c02d5ca5eaeac8906fe31e84547ecc3f224e2: Status 404 returned error can't find the container with id ab6455286795e0e25599c7b0704c02d5ca5eaeac8906fe31e84547ecc3f224e2
Oct 29 22:13:11.055129 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.055092    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:13:11.055238 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.055173    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:13:11.055238 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.055216    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:13:11.149728 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.149699    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"]
Oct 29 22:13:11.150777 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:13:11.150752    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0e7932_be97_4cbb_b6dd_c514b1ba6dcc.slice/crio-38f662168623aa603f410ebd3eb9f89c7bf598a6f18e6df62fa4f161fb3b9cb6 WatchSource:0}: Error finding container 38f662168623aa603f410ebd3eb9f89c7bf598a6f18e6df62fa4f161fb3b9cb6: Status 404 returned error can't find the container with id 38f662168623aa603f410ebd3eb9f89c7bf598a6f18e6df62fa4f161fb3b9cb6
Oct 29 22:13:11.153074 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.153037    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:13:11.153174 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.153114    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:13:11.153174 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.153164    7082 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 29 22:13:11.536473 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.536387    7082 patch_prober.go:29] interesting pod/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 29 22:13:11.536638 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.536468    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 29 22:13:11.626651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.618697    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz" event={"ID":"027d1e1a-2888-45b7-b47e-2b62ca191f7a","Type":"ContainerStarted","Data":"32814cff4a4183d3ce8a8114c365e76cdefaeb5dbad2d2d7190c63cbc07fa82a"}
Oct 29 22:13:11.626651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.618761    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz" event={"ID":"027d1e1a-2888-45b7-b47e-2b62ca191f7a","Type":"ContainerStarted","Data":"ab6455286795e0e25599c7b0704c02d5ca5eaeac8906fe31e84547ecc3f224e2"}
Oct 29 22:13:11.639606 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.639572    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb" event={"ID":"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc","Type":"ContainerStarted","Data":"6ff3769da58d33e970c6e2f32744eb238e97eebefee07dfe880d9b0f660a61f3"}
Oct 29 22:13:11.640514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.640485    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb" event={"ID":"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc","Type":"ContainerStarted","Data":"38f662168623aa603f410ebd3eb9f89c7bf598a6f18e6df62fa4f161fb3b9cb6"}
Oct 29 22:13:11.676494 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.675001    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:11.676494 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.675432    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz" podStartSLOduration=2.675413442 podStartE2EDuration="2.675413442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:11.66782824 +0000 UTC m=+1564.167870535" watchObservedRunningTime="2026-10-29 22:13:11.675413442 +0000 UTC m=+1564.175455742"
Oct 29 22:13:11.734648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.725126    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb" podStartSLOduration=1.725103104 podStartE2EDuration="1.725103104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:11.714171811 +0000 UTC m=+1564.214214111" watchObservedRunningTime="2026-10-29 22:13:11.725103104 +0000 UTC m=+1564.225145404"
Oct 29 22:13:11.775806 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:11.774529    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:12.580849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:12.580812    7082 patch_prober.go:29] interesting pod/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 29 22:13:12.618651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:12.617948    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 29 22:13:12.757435 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:12.711998    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:13:12.757435 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:12.731710    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz" podUID="027d1e1a-2888-45b7-b47e-2b62ca191f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 29 22:13:12.780651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:12.762947    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:13:12.780651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:12.763002    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb" podUID="bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:13:13.190652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.188708    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:13.224959 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.224921    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"]
Oct 29 22:13:13.263066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.263027    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-27cda7-ctfcl"]
Oct 29 22:13:13.314035 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.313992    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": context deadline exceeded"
Oct 29 22:13:13.487871 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.487771    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:13.488452 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.488421    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:13:13.530654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.530486    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:13.544258 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.544218    7082 patch_prober.go:29] interesting pod/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 29 22:13:13.544585 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.544552    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 29 22:13:13.710654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.697489    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e602c944-6a33-49cf-8ab5-1c225fbd4d44" path="/var/lib/kubelet/pods/e602c944-6a33-49cf-8ab5-1c225fbd4d44/volumes"
Oct 29 22:13:13.714650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.712364    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:13:13.714650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.712423    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz" podUID="027d1e1a-2888-45b7-b47e-2b62ca191f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 29 22:13:13.768714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.766338    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:13:13.768714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:13.766409    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb" podUID="bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:13:14.547651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:14.545021    7082 patch_prober.go:29] interesting pod/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Oct 29 22:13:14.547651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:14.545081    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Oct 29 22:13:14.678132 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:14.678103    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:13:14.678360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:14.678339    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz" podUID="027d1e1a-2888-45b7-b47e-2b62ca191f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 29 22:13:14.762992 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:14.762954    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:13:14.763599 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:14.763568    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb" podUID="bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:13:15.674664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:15.674522    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:13:15.675106 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:15.675072    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz" podUID="027d1e1a-2888-45b7-b47e-2b62ca191f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 29 22:13:15.760455 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:15.760404    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:13:15.760647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:15.760476    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb" podUID="bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:13:15.972877 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:15.972801    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:13:15.973917 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:15.973892    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:13:15.974968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:15.974949    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:13:16.675262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:16.675223    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:13:16.675466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:16.675279    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz" podUID="027d1e1a-2888-45b7-b47e-2b62ca191f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 29 22:13:16.760431 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:16.760401    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:13:16.760609 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:16.760447    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb" podUID="bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:13:17.674699 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:17.674671    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:13:17.675197 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:17.675167    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz" podUID="027d1e1a-2888-45b7-b47e-2b62ca191f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 29 22:13:17.759512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:17.759482    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:13:17.759780 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:17.759753    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb" podUID="bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:13:18.689654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:18.684257    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 29 22:13:18.689654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:18.684334    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz" podUID="027d1e1a-2888-45b7-b47e-2b62ca191f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 29 22:13:18.770776 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:18.770737    7082 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body=
Oct 29 22:13:18.771025 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:18.770998    7082 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb" podUID="bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused"
Oct 29 22:13:19.177972 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.177936    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"]
Oct 29 22:13:19.246654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.213714    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"
Oct 29 22:13:19.246654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.237722    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-75db5b\""
Oct 29 22:13:19.246654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.237952    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-75db5b-dockercfg-dq7ld\""
Oct 29 22:13:19.268071 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.267883    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"]
Oct 29 22:13:19.274738 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.274717    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"]
Oct 29 22:13:19.278483 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.278462    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"
Oct 29 22:13:19.280693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.280658    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"]
Oct 29 22:13:19.325691 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.325607    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6xfz\" (UniqueName: \"kubernetes.io/projected/d570bb54-a40a-44f7-afe5-875bc98185b6-kube-api-access-z6xfz\") pod \"payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8\" (UID: \"d570bb54-a40a-44f7-afe5-875bc98185b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"
Oct 29 22:13:19.325847 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.325741    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d570bb54-a40a-44f7-afe5-875bc98185b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8\" (UID: \"d570bb54-a40a-44f7-afe5-875bc98185b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"
Oct 29 22:13:19.325847 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.325770    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e480bf6-b96a-4315-be7e-9180c8173fab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9\" (UID: \"4e480bf6-b96a-4315-be7e-9180c8173fab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"
Oct 29 22:13:19.326029 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.326013    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nfx9\" (UniqueName: \"kubernetes.io/projected/4e480bf6-b96a-4315-be7e-9180c8173fab-kube-api-access-2nfx9\") pod \"payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9\" (UID: \"4e480bf6-b96a-4315-be7e-9180c8173fab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"
Oct 29 22:13:19.428171 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.428046    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nfx9\" (UniqueName: \"kubernetes.io/projected/4e480bf6-b96a-4315-be7e-9180c8173fab-kube-api-access-2nfx9\") pod \"payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9\" (UID: \"4e480bf6-b96a-4315-be7e-9180c8173fab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"
Oct 29 22:13:19.428356 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.428316    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6xfz\" (UniqueName: \"kubernetes.io/projected/d570bb54-a40a-44f7-afe5-875bc98185b6-kube-api-access-z6xfz\") pod \"payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8\" (UID: \"d570bb54-a40a-44f7-afe5-875bc98185b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"
Oct 29 22:13:19.428414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.428402    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d570bb54-a40a-44f7-afe5-875bc98185b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8\" (UID: \"d570bb54-a40a-44f7-afe5-875bc98185b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"
Oct 29 22:13:19.428463 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.428427    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e480bf6-b96a-4315-be7e-9180c8173fab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9\" (UID: \"4e480bf6-b96a-4315-be7e-9180c8173fab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"
Oct 29 22:13:19.429342 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.429314    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d570bb54-a40a-44f7-afe5-875bc98185b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8\" (UID: \"d570bb54-a40a-44f7-afe5-875bc98185b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"
Oct 29 22:13:19.429551 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.429441    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e480bf6-b96a-4315-be7e-9180c8173fab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9\" (UID: \"4e480bf6-b96a-4315-be7e-9180c8173fab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"
Oct 29 22:13:19.451103 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.451063    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nfx9\" (UniqueName: \"kubernetes.io/projected/4e480bf6-b96a-4315-be7e-9180c8173fab-kube-api-access-2nfx9\") pod \"payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9\" (UID: \"4e480bf6-b96a-4315-be7e-9180c8173fab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"
Oct 29 22:13:19.451372 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.451352    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6xfz\" (UniqueName: \"kubernetes.io/projected/d570bb54-a40a-44f7-afe5-875bc98185b6-kube-api-access-z6xfz\") pod \"payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8\" (UID: \"d570bb54-a40a-44f7-afe5-875bc98185b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"
Oct 29 22:13:19.576704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.576672    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"
Oct 29 22:13:19.618768 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:19.618036    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"
Oct 29 22:13:20.042584 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:20.042426    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:20.043805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:20.043782    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:20.049163 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:20.049119    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:13:20.067374 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:20.067350    7082 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:20.068138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:20.068116    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:20.108648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:20.095730    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:13:20.200731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:20.200697    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"]
Oct 29 22:13:20.284119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:20.284084    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9" event={"ID":"4e480bf6-b96a-4315-be7e-9180c8173fab","Type":"ContainerStarted","Data":"ad30c3cbf3be60e9dde51b7430460bf52bbfc2639bcac1b76e9ee0b688ee7054"}
Oct 29 22:13:20.463311 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:20.462776    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"]
Oct 29 22:13:20.507833 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:20.507776    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"]
Oct 29 22:13:20.508305 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:20.508271    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy" containerID="cri-o://1172f637051c7d5c6e26b478bea97d1a88e00b7ffc01afe8fa99a9fec39d0f1d" gracePeriod=30
Oct 29 22:13:21.067863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.067831    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"]
Oct 29 22:13:21.071827 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.071808    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"
Oct 29 22:13:21.087744 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.087723    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:13:21.087915 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.087762    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:13:21.094915 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.094890    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"]
Oct 29 22:13:21.110811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.110459    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:13:21.110811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.110528    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:13:21.200255 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.200213    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h\" (UID: \"9a527702-a704-4c3e-8e19-acf1d5bb9677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"
Oct 29 22:13:21.200447 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.200377    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h\" (UID: \"9a527702-a704-4c3e-8e19-acf1d5bb9677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"
Oct 29 22:13:21.200580 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.200560    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkg94\" (UniqueName: \"kubernetes.io/projected/9a527702-a704-4c3e-8e19-acf1d5bb9677-kube-api-access-qkg94\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h\" (UID: \"9a527702-a704-4c3e-8e19-acf1d5bb9677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"
Oct 29 22:13:21.299462 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.299422    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8" event={"ID":"d570bb54-a40a-44f7-afe5-875bc98185b6","Type":"ContainerStarted","Data":"90768ac43e208e9becc0e83931dc40e98394b37ffc3c73dc57ac9d44475478f0"}
Oct 29 22:13:21.299701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.299519    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8" event={"ID":"d570bb54-a40a-44f7-afe5-875bc98185b6","Type":"ContainerStarted","Data":"a0bf79270d25fde12de765eed9921fe904d3550319ab75ec2ae95d906e602998"}
Oct 29 22:13:21.299701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.299665    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"
Oct 29 22:13:21.301987 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.301956    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9" event={"ID":"4e480bf6-b96a-4315-be7e-9180c8173fab","Type":"ContainerStarted","Data":"49c4029e0ce261218c8b80ccfdf869d1fc8243770bdceec6965b8582d98509b2"}
Oct 29 22:13:21.302121 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.302032    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"
Oct 29 22:13:21.302284 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.302259    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h\" (UID: \"9a527702-a704-4c3e-8e19-acf1d5bb9677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"
Oct 29 22:13:21.302454 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.302439    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h\" (UID: \"9a527702-a704-4c3e-8e19-acf1d5bb9677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"
Oct 29 22:13:21.302710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.302691    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkg94\" (UniqueName: \"kubernetes.io/projected/9a527702-a704-4c3e-8e19-acf1d5bb9677-kube-api-access-qkg94\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h\" (UID: \"9a527702-a704-4c3e-8e19-acf1d5bb9677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"
Oct 29 22:13:21.303333 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.303309    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h\" (UID: \"9a527702-a704-4c3e-8e19-acf1d5bb9677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"
Oct 29 22:13:21.309720 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.305525    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h\" (UID: \"9a527702-a704-4c3e-8e19-acf1d5bb9677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"
Oct 29 22:13:21.325602 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.325570    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkg94\" (UniqueName: \"kubernetes.io/projected/9a527702-a704-4c3e-8e19-acf1d5bb9677-kube-api-access-qkg94\") pod \"maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h\" (UID: \"9a527702-a704-4c3e-8e19-acf1d5bb9677\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"
Oct 29 22:13:21.343511 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.343466    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8" podStartSLOduration=3.343451932 podStartE2EDuration="3.343451932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:21.338861078 +0000 UTC m=+1573.838903378" watchObservedRunningTime="2026-10-29 22:13:21.343451932 +0000 UTC m=+1573.843494230"
Oct 29 22:13:21.388542 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.388457    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"
Oct 29 22:13:21.917983 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.917922    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9" podStartSLOduration=3.9179036739999997 podStartE2EDuration="3.917903674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:21.381662329 +0000 UTC m=+1573.881704629" watchObservedRunningTime="2026-10-29 22:13:21.917903674 +0000 UTC m=+1574.417945974"
Oct 29 22:13:21.920782 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.920755    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"]
Oct 29 22:13:21.921051 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.921026    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2" podUID="a67c489e-b41a-4414-9045-8a435b773168" containerName="payload-processing" containerID="cri-o://cefd1fad79f38d8dfaf8ffb662a63c457613d3da1c6e2cfaadc56519355e0fc2" gracePeriod=30
Oct 29 22:13:21.933698 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:13:21.933669    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a527702_a704_4c3e_8e19_acf1d5bb9677.slice/crio-39e557a9ad8f59d3bf68473f4856fb98b16c3eece2d9a13ba1ddc4b48365c887 WatchSource:0}: Error finding container 39e557a9ad8f59d3bf68473f4856fb98b16c3eece2d9a13ba1ddc4b48365c887: Status 404 returned error can't find the container with id 39e557a9ad8f59d3bf68473f4856fb98b16c3eece2d9a13ba1ddc4b48365c887
Oct 29 22:13:21.939346 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:21.939320    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"]
Oct 29 22:13:22.743319 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:22.720394    7082 generic.go:356] "Generic (PLEG): container finished" podID="a67c489e-b41a-4414-9045-8a435b773168" containerID="cefd1fad79f38d8dfaf8ffb662a63c457613d3da1c6e2cfaadc56519355e0fc2" exitCode=0
Oct 29 22:13:22.743319 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:22.720622    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2" event={"ID":"a67c489e-b41a-4414-9045-8a435b773168","Type":"ContainerDied","Data":"cefd1fad79f38d8dfaf8ffb662a63c457613d3da1c6e2cfaadc56519355e0fc2"}
Oct 29 22:13:22.743319 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:22.729077    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"]
Oct 29 22:13:22.753302 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:22.752393    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h" event={"ID":"9a527702-a704-4c3e-8e19-acf1d5bb9677","Type":"ContainerStarted","Data":"39e557a9ad8f59d3bf68473f4856fb98b16c3eece2d9a13ba1ddc4b48365c887"}
Oct 29 22:13:22.763264 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:22.763238    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"
Oct 29 22:13:22.764748 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:22.764723    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f82846-hxlj6"]
Oct 29 22:13:22.840244 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:22.838062    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67c489e-b41a-4414-9045-8a435b773168-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67c489e-b41a-4414-9045-8a435b773168-plugins-config-volume\") pod \"a67c489e-b41a-4414-9045-8a435b773168\" (UID: \"a67c489e-b41a-4414-9045-8a435b773168\") "
Oct 29 22:13:22.840244 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:22.838298    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67c489e-b41a-4414-9045-8a435b773168-kube-api-access-6pkh8\" (UniqueName: \"kubernetes.io/projected/a67c489e-b41a-4414-9045-8a435b773168-kube-api-access-6pkh8\") pod \"a67c489e-b41a-4414-9045-8a435b773168\" (UID: \"a67c489e-b41a-4414-9045-8a435b773168\") "
Oct 29 22:13:22.841040 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:22.840707    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67c489e-b41a-4414-9045-8a435b773168-plugins-config-volume" pod "a67c489e-b41a-4414-9045-8a435b773168" (UID: "a67c489e-b41a-4414-9045-8a435b773168"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:22.950406 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:22.942034    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67c489e-b41a-4414-9045-8a435b773168-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:22.999591 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:22.992820    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67c489e-b41a-4414-9045-8a435b773168-kube-api-access-6pkh8" pod "a67c489e-b41a-4414-9045-8a435b773168" (UID: "a67c489e-b41a-4414-9045-8a435b773168"). InnerVolumeSpecName "kube-api-access-6pkh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:23.044084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:23.044043    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pkh8\" (UniqueName: \"kubernetes.io/projected/a67c489e-b41a-4414-9045-8a435b773168-kube-api-access-6pkh8\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:23.636150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:23.636105    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4095f5b-db7d-41c5-a147-0ac1b6d881a2" path="/var/lib/kubelet/pods/a4095f5b-db7d-41c5-a147-0ac1b6d881a2/volumes"
Oct 29 22:13:23.763718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:23.763681    7082 generic.go:356] "Generic (PLEG): container finished" podID="9a527702-a704-4c3e-8e19-acf1d5bb9677" containerID="5a5654f674607433663955d3d3e1b341b5dccfc88d8e1ca3cd8dc19eb69838db" exitCode=0
Oct 29 22:13:23.764164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:23.763764    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h" event={"ID":"9a527702-a704-4c3e-8e19-acf1d5bb9677","Type":"ContainerDied","Data":"5a5654f674607433663955d3d3e1b341b5dccfc88d8e1ca3cd8dc19eb69838db"}
Oct 29 22:13:23.765711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:23.765584    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2" event={"ID":"a67c489e-b41a-4414-9045-8a435b773168","Type":"ContainerDied","Data":"92abdd5896e3369a83fe21f4cb9390538c877bd7e6e04ca73baff9906f210817"}
Oct 29 22:13:23.765856 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:23.765763    7082 scope.go:122] "RemoveContainer" containerID="cefd1fad79f38d8dfaf8ffb662a63c457613d3da1c6e2cfaadc56519355e0fc2"
Oct 29 22:13:23.765856 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:23.765610    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"
Oct 29 22:13:23.846716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:23.846684    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"]
Oct 29 22:13:23.851610 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:23.851588    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f82846-776b6998c6-vzxz2"]
Oct 29 22:13:24.958354 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:24.958332    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"
Oct 29 22:13:25.081522 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.081493    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-xks-ca\") pod \"9a527702-a704-4c3e-8e19-acf1d5bb9677\" (UID: \"9a527702-a704-4c3e-8e19-acf1d5bb9677\") "
Oct 29 22:13:25.081717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.081709    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a527702-a704-4c3e-8e19-acf1d5bb9677-kube-api-access-qkg94\" (UniqueName: \"kubernetes.io/projected/9a527702-a704-4c3e-8e19-acf1d5bb9677-kube-api-access-qkg94\") pod \"9a527702-a704-4c3e-8e19-acf1d5bb9677\" (UID: \"9a527702-a704-4c3e-8e19-acf1d5bb9677\") "
Oct 29 22:13:25.081852 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.081839    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-openshift-ca\") pod \"9a527702-a704-4c3e-8e19-acf1d5bb9677\" (UID: \"9a527702-a704-4c3e-8e19-acf1d5bb9677\") "
Oct 29 22:13:25.082262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.082191    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-openshift-ca" pod "9a527702-a704-4c3e-8e19-acf1d5bb9677" (UID: "9a527702-a704-4c3e-8e19-acf1d5bb9677"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:25.082621 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.082602    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:25.083877 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.083849    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-xks-ca" pod "9a527702-a704-4c3e-8e19-acf1d5bb9677" (UID: "9a527702-a704-4c3e-8e19-acf1d5bb9677"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:25.083962 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.083894    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a527702-a704-4c3e-8e19-acf1d5bb9677-kube-api-access-qkg94" pod "9a527702-a704-4c3e-8e19-acf1d5bb9677" (UID: "9a527702-a704-4c3e-8e19-acf1d5bb9677"). InnerVolumeSpecName "kube-api-access-qkg94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:25.184657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.184609    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkg94\" (UniqueName: \"kubernetes.io/projected/9a527702-a704-4c3e-8e19-acf1d5bb9677-kube-api-access-qkg94\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:25.184657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.184662    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a527702-a704-4c3e-8e19-acf1d5bb9677-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:25.636283 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.636253    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67c489e-b41a-4414-9045-8a435b773168" path="/var/lib/kubelet/pods/a67c489e-b41a-4414-9045-8a435b773168/volumes"
Oct 29 22:13:25.777871 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.777840    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:13:25.788161 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.788092    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h" event={"ID":"9a527702-a704-4c3e-8e19-acf1d5bb9677","Type":"ContainerDied","Data":"39e557a9ad8f59d3bf68473f4856fb98b16c3eece2d9a13ba1ddc4b48365c887"}
Oct 29 22:13:25.788161 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.788105    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"
Oct 29 22:13:25.788161 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.788146    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39e557a9ad8f59d3bf68473f4856fb98b16c3eece2d9a13ba1ddc4b48365c887"
Oct 29 22:13:25.790363 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.790311    7082 generic.go:356] "Generic (PLEG): container finished" podID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerID="1172f637051c7d5c6e26b478bea97d1a88e00b7ffc01afe8fa99a9fec39d0f1d" exitCode=0
Oct 29 22:13:25.790509 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.790388    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" event={"ID":"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248","Type":"ContainerDied","Data":"1172f637051c7d5c6e26b478bea97d1a88e00b7ffc01afe8fa99a9fec39d0f1d"}
Oct 29 22:13:25.790509 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.790440    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5" event={"ID":"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248","Type":"ContainerDied","Data":"20a39401edad6ecf416f8188fcaa236d7d185b1c98b2e9d55ba8b102b5224a33"}
Oct 29 22:13:25.790509 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.790471    7082 scope.go:122] "RemoveContainer" containerID="1172f637051c7d5c6e26b478bea97d1a88e00b7ffc01afe8fa99a9fec39d0f1d"
Oct 29 22:13:25.791055 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.791038    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"
Oct 29 22:13:25.802178 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.802147    7082 scope.go:122] "RemoveContainer" containerID="5e5971e8c0ee066cbbaa007def83ced34ee096d0d8d2375740f7302916d71340"
Oct 29 22:13:25.813943 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.813920    7082 scope.go:122] "RemoveContainer" containerID="1172f637051c7d5c6e26b478bea97d1a88e00b7ffc01afe8fa99a9fec39d0f1d"
Oct 29 22:13:25.814241 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:13:25.814221    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1172f637051c7d5c6e26b478bea97d1a88e00b7ffc01afe8fa99a9fec39d0f1d\": container with ID starting with 1172f637051c7d5c6e26b478bea97d1a88e00b7ffc01afe8fa99a9fec39d0f1d not found: ID does not exist" containerID="1172f637051c7d5c6e26b478bea97d1a88e00b7ffc01afe8fa99a9fec39d0f1d"
Oct 29 22:13:25.814308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.814249    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1172f637051c7d5c6e26b478bea97d1a88e00b7ffc01afe8fa99a9fec39d0f1d"} err="failed to get container status \"1172f637051c7d5c6e26b478bea97d1a88e00b7ffc01afe8fa99a9fec39d0f1d\": rpc error: code = NotFound desc = could not find container \"1172f637051c7d5c6e26b478bea97d1a88e00b7ffc01afe8fa99a9fec39d0f1d\": container with ID starting with 1172f637051c7d5c6e26b478bea97d1a88e00b7ffc01afe8fa99a9fec39d0f1d not found: ID does not exist"
Oct 29 22:13:25.814308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.814266    7082 scope.go:122] "RemoveContainer" containerID="5e5971e8c0ee066cbbaa007def83ced34ee096d0d8d2375740f7302916d71340"
Oct 29 22:13:25.814536 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:13:25.814520    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5971e8c0ee066cbbaa007def83ced34ee096d0d8d2375740f7302916d71340\": container with ID starting with 5e5971e8c0ee066cbbaa007def83ced34ee096d0d8d2375740f7302916d71340 not found: ID does not exist" containerID="5e5971e8c0ee066cbbaa007def83ced34ee096d0d8d2375740f7302916d71340"
Oct 29 22:13:25.814593 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.814541    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5971e8c0ee066cbbaa007def83ced34ee096d0d8d2375740f7302916d71340"} err="failed to get container status \"5e5971e8c0ee066cbbaa007def83ced34ee096d0d8d2375740f7302916d71340\": rpc error: code = NotFound desc = could not find container \"5e5971e8c0ee066cbbaa007def83ced34ee096d0d8d2375740f7302916d71340\": container with ID starting with 5e5971e8c0ee066cbbaa007def83ced34ee096d0d8d2375740f7302916d71340 not found: ID does not exist"
Oct 29 22:13:25.899866 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.899828    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-data\") pod \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") "
Oct 29 22:13:25.900053 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.899887    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-socket\") pod \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") "
Oct 29 22:13:25.900053 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.899920    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-podinfo\") pod \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") "
Oct 29 22:13:25.900053 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.899975    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-envoy\") pod \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") "
Oct 29 22:13:25.900053 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.900011    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-token\" (UniqueName: \"kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-token\") pod \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") "
Oct 29 22:13:25.900315 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.900067    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-certs\") pod \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") "
Oct 29 22:13:25.900315 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.900104    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-data" pod "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" (UID: "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:25.900315 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.900180    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-socket" pod "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" (UID: "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:25.900315 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.900220    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-kube-api-access-2k8fz\" (UniqueName: \"kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-kube-api-access-2k8fz\") pod \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") "
Oct 29 22:13:25.900537 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.900331    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istiod-ca-cert\") pod \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") "
Oct 29 22:13:25.900537 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.900386    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-credential-socket\") pod \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\" (UID: \"e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248\") "
Oct 29 22:13:25.900537 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.900459    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-certs" pod "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" (UID: "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:25.900899 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.900875    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istiod-ca-cert" pod "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" (UID: "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:25.901142 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.901117    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-credential-socket" pod "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" (UID: "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:25.901860 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.901836    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:25.901958 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.901867    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:25.901958 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.901883    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:25.901958 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.901898    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:25.901958 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.901911    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:25.903027 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.903004    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-envoy" pod "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" (UID: "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:25.903124 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.903105    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-token" pod "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" (UID: "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:25.903179 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.903123    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-podinfo" pod "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" (UID: "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:25.903179 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:25.903133    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-kube-api-access-2k8fz" pod "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" (UID: "e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248"). InnerVolumeSpecName "kube-api-access-2k8fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:26.003467 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:26.003439    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:26.003796 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:26.003469    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:26.003796 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:26.003503    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:26.003796 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:26.003517    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k8fz\" (UniqueName: \"kubernetes.io/projected/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248-kube-api-access-2k8fz\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:26.141345 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:26.141314    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"]
Oct 29 22:13:26.147000 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:26.146972    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-27cda7-openshift-default-86df67d6bf-8dkr5"]
Oct 29 22:13:27.635074 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:27.635041    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" path="/var/lib/kubelet/pods/e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248/volumes"
Oct 29 22:13:28.414464 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:13:28.414424    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:13:30.414968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:30.414309    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"]
Oct 29 22:13:30.414968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:30.414656    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp" podUID="0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" containerName="istio-proxy" containerID="cri-o://1344d28f560ef40d94601ccbe58d0de0bc1823751a598b10a4802b0af034e1d0" gracePeriod=30
Oct 29 22:13:30.421992 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:30.421964    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"]
Oct 29 22:13:30.422270 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:30.422239    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc" podUID="edd69823-b544-4f2b-90e8-08ed1a7c00d9" containerName="istio-proxy" containerID="cri-o://79a65f106c71393b77c61138cb0fbae9b5a7936ed2e116a4adca53791c194dd1" gracePeriod=30
Oct 29 22:13:31.166111 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.166076    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"]
Oct 29 22:13:31.166992 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.166967    7082 cpu_manager.go:414] "containerMap: removing container" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy"
Oct 29 22:13:31.166992 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.166990    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy"
Oct 29 22:13:31.167185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.167011    7082 cpu_manager.go:414] "containerMap: removing container" podUID="a67c489e-b41a-4414-9045-8a435b773168" containerName="payload-processing"
Oct 29 22:13:31.167185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.167018    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67c489e-b41a-4414-9045-8a435b773168" containerName="payload-processing"
Oct 29 22:13:31.167185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.167061    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9a527702-a704-4c3e-8e19-acf1d5bb9677" containerName="revoke-keys"
Oct 29 22:13:31.167185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.167069    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a527702-a704-4c3e-8e19-acf1d5bb9677" containerName="revoke-keys"
Oct 29 22:13:31.167185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.167085    7082 cpu_manager.go:414] "containerMap: removing container" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy"
Oct 29 22:13:31.167185 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.167091    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy"
Oct 29 22:13:31.167543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.167294    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy"
Oct 29 22:13:31.167543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.167309    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67c489e-b41a-4414-9045-8a435b773168" containerName="payload-processing"
Oct 29 22:13:31.167543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.167325    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a527702-a704-4c3e-8e19-acf1d5bb9677" containerName="revoke-keys"
Oct 29 22:13:31.167543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.167336    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a92ac5-7e2c-46c6-9d6f-6ffdb3156248" containerName="istio-proxy"
Oct 29 22:13:31.172140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.172118    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"
Oct 29 22:13:31.192085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.192038    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"]
Oct 29 22:13:31.196820 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.196798    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:13:31.196820 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.196803    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:13:31.209727 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.209696    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:13:31.209870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.209736    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:13:31.285752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.285720    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg\" (UID: \"4ee3457a-f692-40c5-8762-02f3141561ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"
Oct 29 22:13:31.285938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.285821    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smjvb\" (UniqueName: \"kubernetes.io/projected/4ee3457a-f692-40c5-8762-02f3141561ef-kube-api-access-smjvb\") pod \"maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg\" (UID: \"4ee3457a-f692-40c5-8762-02f3141561ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"
Oct 29 22:13:31.286037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.286013    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg\" (UID: \"4ee3457a-f692-40c5-8762-02f3141561ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"
Oct 29 22:13:31.387805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.387770    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg\" (UID: \"4ee3457a-f692-40c5-8762-02f3141561ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"
Oct 29 22:13:31.387971 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.387816    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smjvb\" (UniqueName: \"kubernetes.io/projected/4ee3457a-f692-40c5-8762-02f3141561ef-kube-api-access-smjvb\") pod \"maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg\" (UID: \"4ee3457a-f692-40c5-8762-02f3141561ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"
Oct 29 22:13:31.387971 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.387917    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg\" (UID: \"4ee3457a-f692-40c5-8762-02f3141561ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"
Oct 29 22:13:31.388594 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.388569    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg\" (UID: \"4ee3457a-f692-40c5-8762-02f3141561ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"
Oct 29 22:13:31.390348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.390323    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg\" (UID: \"4ee3457a-f692-40c5-8762-02f3141561ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"
Oct 29 22:13:31.411023 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.411002    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smjvb\" (UniqueName: \"kubernetes.io/projected/4ee3457a-f692-40c5-8762-02f3141561ef-kube-api-access-smjvb\") pod \"maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg\" (UID: \"4ee3457a-f692-40c5-8762-02f3141561ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"
Oct 29 22:13:31.487582 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.487499    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"
Oct 29 22:13:31.917396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:31.915844    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"]
Oct 29 22:13:32.755259 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:32.755182    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"
Oct 29 22:13:32.755619 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:32.755261    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"
Oct 29 22:13:32.932963 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:32.932926    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg" event={"ID":"4ee3457a-f692-40c5-8762-02f3141561ef","Type":"ContainerStarted","Data":"e1886b439d28f9f997e9e847ab871fbb653f8a0f989a89b794d327f295d6157a"}
Oct 29 22:13:32.933183 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:32.932990    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg" event={"ID":"4ee3457a-f692-40c5-8762-02f3141561ef","Type":"ContainerStarted","Data":"508a1654675cb79655916b6ae50f91d4192658b8a9e7f3fae190539247ba9324"}
Oct 29 22:13:32.980203 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:32.980147    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg" podStartSLOduration=1.980133188 podStartE2EDuration="1.980133188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:32.976859992 +0000 UTC m=+1585.476902290" watchObservedRunningTime="2026-10-29 22:13:32.980133188 +0000 UTC m=+1585.480175540"
Oct 29 22:13:33.944561 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:33.944522    7082 generic.go:356] "Generic (PLEG): container finished" podID="4ee3457a-f692-40c5-8762-02f3141561ef" containerID="e1886b439d28f9f997e9e847ab871fbb653f8a0f989a89b794d327f295d6157a" exitCode=0
Oct 29 22:13:33.944974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:33.944563    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg" event={"ID":"4ee3457a-f692-40c5-8762-02f3141561ef","Type":"ContainerDied","Data":"e1886b439d28f9f997e9e847ab871fbb653f8a0f989a89b794d327f295d6157a"}
Oct 29 22:13:35.145112 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.145085    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"
Oct 29 22:13:35.276166 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.276083    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee3457a-f692-40c5-8762-02f3141561ef-kube-api-access-smjvb\" (UniqueName: \"kubernetes.io/projected/4ee3457a-f692-40c5-8762-02f3141561ef-kube-api-access-smjvb\") pod \"4ee3457a-f692-40c5-8762-02f3141561ef\" (UID: \"4ee3457a-f692-40c5-8762-02f3141561ef\") "
Oct 29 22:13:35.276324 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.276214    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-xks-ca\") pod \"4ee3457a-f692-40c5-8762-02f3141561ef\" (UID: \"4ee3457a-f692-40c5-8762-02f3141561ef\") "
Oct 29 22:13:35.276502 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.276477    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-openshift-ca\") pod \"4ee3457a-f692-40c5-8762-02f3141561ef\" (UID: \"4ee3457a-f692-40c5-8762-02f3141561ef\") "
Oct 29 22:13:35.276865 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.276839    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-openshift-ca" pod "4ee3457a-f692-40c5-8762-02f3141561ef" (UID: "4ee3457a-f692-40c5-8762-02f3141561ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:35.277217 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.277197    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:35.278451 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.278427    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee3457a-f692-40c5-8762-02f3141561ef-kube-api-access-smjvb" pod "4ee3457a-f692-40c5-8762-02f3141561ef" (UID: "4ee3457a-f692-40c5-8762-02f3141561ef"). InnerVolumeSpecName "kube-api-access-smjvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:35.278593 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.278575    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-xks-ca" pod "4ee3457a-f692-40c5-8762-02f3141561ef" (UID: "4ee3457a-f692-40c5-8762-02f3141561ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:35.379231 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.379195    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smjvb\" (UniqueName: \"kubernetes.io/projected/4ee3457a-f692-40c5-8762-02f3141561ef-kube-api-access-smjvb\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:35.379231 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.379229    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee3457a-f692-40c5-8762-02f3141561ef-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:35.708688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.708659    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:13:35.775959 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.775929    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.957277    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-envoy\") pod \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.957329    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-credential-socket\") pod \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.957394    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istiod-ca-cert\") pod \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.957453    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-credential-socket\") pod \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.957533    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-certs\") pod \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.957584    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-token\" (UniqueName: \"kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-token\") pod \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.957613    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-token\" (UniqueName: \"kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-token\") pod \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.957701    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-kube-api-access-496b4\" (UniqueName: \"kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-kube-api-access-496b4\") pod \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.957756    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-certs\") pod \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.957816    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-podinfo\") pod \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.957991    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istiod-ca-cert\") pod \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.958059    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-podinfo\") pod \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.958206    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-data\") pod \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.958240    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-socket\") pod \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.958270    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-socket\") pod \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.958343    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-kube-api-access-mnl8c\" (UniqueName: \"kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-kube-api-access-mnl8c\") pod \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\" (UID: \"edd69823-b544-4f2b-90e8-08ed1a7c00d9\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.958382    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-data\") pod \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") "
Oct 29 22:13:35.960652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.958444    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-envoy\") pod \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\" (UID: \"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10\") "
Oct 29 22:13:36.053676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.985744    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-certs" pod "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" (UID: "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.053676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.990989    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istiod-ca-cert" pod "edd69823-b544-4f2b-90e8-08ed1a7c00d9" (UID: "edd69823-b544-4f2b-90e8-08ed1a7c00d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:36.053676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.995874    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-credential-socket" pod "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" (UID: "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.053676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.998127    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-credential-socket" pod "edd69823-b544-4f2b-90e8-08ed1a7c00d9" (UID: "edd69823-b544-4f2b-90e8-08ed1a7c00d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.053676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:35.998224    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-certs" pod "edd69823-b544-4f2b-90e8-08ed1a7c00d9" (UID: "edd69823-b544-4f2b-90e8-08ed1a7c00d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.053676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.001035    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istiod-ca-cert" pod "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" (UID: "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:36.053676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.034212    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-envoy" pod "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" (UID: "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.053676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.034482    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-socket" pod "edd69823-b544-4f2b-90e8-08ed1a7c00d9" (UID: "edd69823-b544-4f2b-90e8-08ed1a7c00d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.053676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.034684    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-data" pod "edd69823-b544-4f2b-90e8-08ed1a7c00d9" (UID: "edd69823-b544-4f2b-90e8-08ed1a7c00d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.053676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.034879    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-socket" pod "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" (UID: "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.105041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.067318    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-data" pod "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" (UID: "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.105041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.074418    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.105041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.074446    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.105041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.074461    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.105041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.074474    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.105041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.074488    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.105041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.074501    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.105041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.074514    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.105041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.074531    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.105041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.074545    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.105041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.074557    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.105041 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.074570    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.152729 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.144957    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-podinfo" pod "edd69823-b544-4f2b-90e8-08ed1a7c00d9" (UID: "edd69823-b544-4f2b-90e8-08ed1a7c00d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:36.226063 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.176236    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-token" pod "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" (UID: "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:36.226063 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.213957    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.226063 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.213994    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.235694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.226718    7082 generic.go:356] "Generic (PLEG): container finished" podID="0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" containerID="1344d28f560ef40d94601ccbe58d0de0bc1823751a598b10a4802b0af034e1d0" exitCode=0
Oct 29 22:13:36.235694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.226920    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp" event={"ID":"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10","Type":"ContainerDied","Data":"1344d28f560ef40d94601ccbe58d0de0bc1823751a598b10a4802b0af034e1d0"}
Oct 29 22:13:36.235694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.226974    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp" event={"ID":"0ffa529b-4b5a-4dda-a6d4-64f1566ccf10","Type":"ContainerDied","Data":"57863ea8fb151b32ce4ad0c5d0f7c1bc683a8a360440c1c40570504801812db8"}
Oct 29 22:13:36.235694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.227003    7082 scope.go:122] "RemoveContainer" containerID="1344d28f560ef40d94601ccbe58d0de0bc1823751a598b10a4802b0af034e1d0"
Oct 29 22:13:36.235694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.227204    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"
Oct 29 22:13:36.273687 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.271829    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-kube-api-access-mnl8c" pod "edd69823-b544-4f2b-90e8-08ed1a7c00d9" (UID: "edd69823-b544-4f2b-90e8-08ed1a7c00d9"). InnerVolumeSpecName "kube-api-access-mnl8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:36.299994 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.299956    7082 generic.go:356] "Generic (PLEG): container finished" podID="edd69823-b544-4f2b-90e8-08ed1a7c00d9" containerID="79a65f106c71393b77c61138cb0fbae9b5a7936ed2e116a4adca53791c194dd1" exitCode=0
Oct 29 22:13:36.310910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.300411    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"
Oct 29 22:13:36.323698 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.300434    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc" event={"ID":"edd69823-b544-4f2b-90e8-08ed1a7c00d9","Type":"ContainerDied","Data":"79a65f106c71393b77c61138cb0fbae9b5a7936ed2e116a4adca53791c194dd1"}
Oct 29 22:13:36.407420 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.406162    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"
Oct 29 22:13:36.408606 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.408569    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc" event={"ID":"edd69823-b544-4f2b-90e8-08ed1a7c00d9","Type":"ContainerDied","Data":"11a6f5dd7ad379eb25e1240024b9d383de13f9990d06929550abe0618653e547"}
Oct 29 22:13:36.408894 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.408860    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg" event={"ID":"4ee3457a-f692-40c5-8762-02f3141561ef","Type":"ContainerDied","Data":"508a1654675cb79655916b6ae50f91d4192658b8a9e7f3fae190539247ba9324"}
Oct 29 22:13:36.409058 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.409042    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="508a1654675cb79655916b6ae50f91d4192658b8a9e7f3fae190539247ba9324"
Oct 29 22:13:36.413158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.413127    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-token" pod "edd69823-b544-4f2b-90e8-08ed1a7c00d9" (UID: "edd69823-b544-4f2b-90e8-08ed1a7c00d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:36.431660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.427749    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnl8c\" (UniqueName: \"kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-kube-api-access-mnl8c\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.464889 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.464827    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-kube-api-access-496b4" pod "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" (UID: "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10"). InnerVolumeSpecName "kube-api-access-496b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:36.465805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.465760    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-envoy" pod "edd69823-b544-4f2b-90e8-08ed1a7c00d9" (UID: "edd69823-b544-4f2b-90e8-08ed1a7c00d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.535979 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.535485    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-podinfo" pod "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" (UID: "0ffa529b-4b5a-4dda-a6d4-64f1566ccf10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:36.571991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.557394    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.571991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.557436    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.571991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.557452    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edd69823-b544-4f2b-90e8-08ed1a7c00d9-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.571991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.557466    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-496b4\" (UniqueName: \"kubernetes.io/projected/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10-kube-api-access-496b4\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.599356 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.599331    7082 scope.go:122] "RemoveContainer" containerID="1344d28f560ef40d94601ccbe58d0de0bc1823751a598b10a4802b0af034e1d0"
Oct 29 22:13:36.599831 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:13:36.599805    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1344d28f560ef40d94601ccbe58d0de0bc1823751a598b10a4802b0af034e1d0\": container with ID starting with 1344d28f560ef40d94601ccbe58d0de0bc1823751a598b10a4802b0af034e1d0 not found: ID does not exist" containerID="1344d28f560ef40d94601ccbe58d0de0bc1823751a598b10a4802b0af034e1d0"
Oct 29 22:13:36.599894 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.599846    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1344d28f560ef40d94601ccbe58d0de0bc1823751a598b10a4802b0af034e1d0"} err="failed to get container status \"1344d28f560ef40d94601ccbe58d0de0bc1823751a598b10a4802b0af034e1d0\": rpc error: code = NotFound desc = could not find container \"1344d28f560ef40d94601ccbe58d0de0bc1823751a598b10a4802b0af034e1d0\": container with ID starting with 1344d28f560ef40d94601ccbe58d0de0bc1823751a598b10a4802b0af034e1d0 not found: ID does not exist"
Oct 29 22:13:36.599894 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.599870    7082 scope.go:122] "RemoveContainer" containerID="79a65f106c71393b77c61138cb0fbae9b5a7936ed2e116a4adca53791c194dd1"
Oct 29 22:13:36.618471 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.618450    7082 scope.go:122] "RemoveContainer" containerID="79a65f106c71393b77c61138cb0fbae9b5a7936ed2e116a4adca53791c194dd1"
Oct 29 22:13:36.619081 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:13:36.619052    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a65f106c71393b77c61138cb0fbae9b5a7936ed2e116a4adca53791c194dd1\": container with ID starting with 79a65f106c71393b77c61138cb0fbae9b5a7936ed2e116a4adca53791c194dd1 not found: ID does not exist" containerID="79a65f106c71393b77c61138cb0fbae9b5a7936ed2e116a4adca53791c194dd1"
Oct 29 22:13:36.619210 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.619088    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a65f106c71393b77c61138cb0fbae9b5a7936ed2e116a4adca53791c194dd1"} err="failed to get container status \"79a65f106c71393b77c61138cb0fbae9b5a7936ed2e116a4adca53791c194dd1\": rpc error: code = NotFound desc = could not find container \"79a65f106c71393b77c61138cb0fbae9b5a7936ed2e116a4adca53791c194dd1\": container with ID starting with 79a65f106c71393b77c61138cb0fbae9b5a7936ed2e116a4adca53791c194dd1 not found: ID does not exist"
Oct 29 22:13:36.631899 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.631873    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"]
Oct 29 22:13:36.642797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.642775    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-mhmxp"]
Oct 29 22:13:36.666670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.666639    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"]
Oct 29 22:13:36.670608 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.670584    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-zlrtc"]
Oct 29 22:13:36.818889 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.818856    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"]
Oct 29 22:13:36.823078 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:36.823046    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-48c52b-zkxdt"]
Oct 29 22:13:37.634638 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:37.634602    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" path="/var/lib/kubelet/pods/0ffa529b-4b5a-4dda-a6d4-64f1566ccf10/volumes"
Oct 29 22:13:37.635190 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:37.635172    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1461211-ff84-45be-b637-0e452114b0bf" path="/var/lib/kubelet/pods/b1461211-ff84-45be-b637-0e452114b0bf/volumes"
Oct 29 22:13:37.635741 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:37.635723    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edd69823-b544-4f2b-90e8-08ed1a7c00d9" path="/var/lib/kubelet/pods/edd69823-b544-4f2b-90e8-08ed1a7c00d9/volumes"
Oct 29 22:13:38.471035 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.470999    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"]
Oct 29 22:13:38.471762 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.471747    7082 cpu_manager.go:414] "containerMap: removing container" podUID="4ee3457a-f692-40c5-8762-02f3141561ef" containerName="revoke-keys"
Oct 29 22:13:38.471820 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.471763    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee3457a-f692-40c5-8762-02f3141561ef" containerName="revoke-keys"
Oct 29 22:13:38.471820 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.471796    7082 cpu_manager.go:414] "containerMap: removing container" podUID="edd69823-b544-4f2b-90e8-08ed1a7c00d9" containerName="istio-proxy"
Oct 29 22:13:38.471820 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.471801    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edd69823-b544-4f2b-90e8-08ed1a7c00d9" containerName="istio-proxy"
Oct 29 22:13:38.471923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.471828    7082 cpu_manager.go:414] "containerMap: removing container" podUID="0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" containerName="istio-proxy"
Oct 29 22:13:38.471923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.471834    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" containerName="istio-proxy"
Oct 29 22:13:38.471988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.471933    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee3457a-f692-40c5-8762-02f3141561ef" containerName="revoke-keys"
Oct 29 22:13:38.471988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.471946    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="edd69823-b544-4f2b-90e8-08ed1a7c00d9" containerName="istio-proxy"
Oct 29 22:13:38.471988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.471958    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ffa529b-4b5a-4dda-a6d4-64f1566ccf10" containerName="istio-proxy"
Oct 29 22:13:38.477469 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.477451    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"
Oct 29 22:13:38.498497 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.498472    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"]
Oct 29 22:13:38.504884 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.504029    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:13:38.504884 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.504707    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:13:38.504884 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.504799    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:13:38.504884 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.504858    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:13:38.522054 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.522029    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx\" (UID: \"9a169dae-6fcb-4ce3-8132-b00da958bd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"
Oct 29 22:13:38.522254 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.522231    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9g6j\" (UniqueName: \"kubernetes.io/projected/9a169dae-6fcb-4ce3-8132-b00da958bd19-kube-api-access-v9g6j\") pod \"maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx\" (UID: \"9a169dae-6fcb-4ce3-8132-b00da958bd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"
Oct 29 22:13:38.522508 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.522489    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx\" (UID: \"9a169dae-6fcb-4ce3-8132-b00da958bd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"
Oct 29 22:13:38.624145 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.624114    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx\" (UID: \"9a169dae-6fcb-4ce3-8132-b00da958bd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"
Oct 29 22:13:38.624323 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.624231    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9g6j\" (UniqueName: \"kubernetes.io/projected/9a169dae-6fcb-4ce3-8132-b00da958bd19-kube-api-access-v9g6j\") pod \"maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx\" (UID: \"9a169dae-6fcb-4ce3-8132-b00da958bd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"
Oct 29 22:13:38.624462 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.624441    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx\" (UID: \"9a169dae-6fcb-4ce3-8132-b00da958bd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"
Oct 29 22:13:38.624849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.624829    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx\" (UID: \"9a169dae-6fcb-4ce3-8132-b00da958bd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"
Oct 29 22:13:38.627157 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.627130    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx\" (UID: \"9a169dae-6fcb-4ce3-8132-b00da958bd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"
Oct 29 22:13:38.660650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.657690    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9g6j\" (UniqueName: \"kubernetes.io/projected/9a169dae-6fcb-4ce3-8132-b00da958bd19-kube-api-access-v9g6j\") pod \"maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx\" (UID: \"9a169dae-6fcb-4ce3-8132-b00da958bd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"
Oct 29 22:13:38.795217 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:38.795118    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"
Oct 29 22:13:39.122228 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:39.122201    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"]
Oct 29 22:13:39.123813 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:13:39.123769    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a169dae_6fcb_4ce3_8132_b00da958bd19.slice/crio-2c1b061c97e3551120c04afffdb14931898a0d85fb115a18319e85ddecd63b72 WatchSource:0}: Error finding container 2c1b061c97e3551120c04afffdb14931898a0d85fb115a18319e85ddecd63b72: Status 404 returned error can't find the container with id 2c1b061c97e3551120c04afffdb14931898a0d85fb115a18319e85ddecd63b72
Oct 29 22:13:39.437986 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:39.437884    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx" event={"ID":"9a169dae-6fcb-4ce3-8132-b00da958bd19","Type":"ContainerStarted","Data":"2c1b061c97e3551120c04afffdb14931898a0d85fb115a18319e85ddecd63b72"}
Oct 29 22:13:40.464943 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:40.464849    7082 generic.go:356] "Generic (PLEG): container finished" podID="9a169dae-6fcb-4ce3-8132-b00da958bd19" containerID="fbfdb3ee3604bbe0c8785c4c70868c45270fb6b846a04a8e1b9b7aee685e8a7f" exitCode=0
Oct 29 22:13:40.466109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:40.466078    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx" event={"ID":"9a169dae-6fcb-4ce3-8132-b00da958bd19","Type":"ContainerDied","Data":"fbfdb3ee3604bbe0c8785c4c70868c45270fb6b846a04a8e1b9b7aee685e8a7f"}
Oct 29 22:13:41.614406 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:41.614366    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"]
Oct 29 22:13:41.645909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:41.645877    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"]
Oct 29 22:13:41.660249 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:41.660222    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"
Oct 29 22:13:41.785418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:41.785333    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a169dae-6fcb-4ce3-8132-b00da958bd19-kube-api-access-v9g6j\" (UniqueName: \"kubernetes.io/projected/9a169dae-6fcb-4ce3-8132-b00da958bd19-kube-api-access-v9g6j\") pod \"9a169dae-6fcb-4ce3-8132-b00da958bd19\" (UID: \"9a169dae-6fcb-4ce3-8132-b00da958bd19\") "
Oct 29 22:13:41.785597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:41.785486    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-xks-ca\") pod \"9a169dae-6fcb-4ce3-8132-b00da958bd19\" (UID: \"9a169dae-6fcb-4ce3-8132-b00da958bd19\") "
Oct 29 22:13:41.785597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:41.785554    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-openshift-ca\") pod \"9a169dae-6fcb-4ce3-8132-b00da958bd19\" (UID: \"9a169dae-6fcb-4ce3-8132-b00da958bd19\") "
Oct 29 22:13:41.786517 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:41.786489    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-openshift-ca" pod "9a169dae-6fcb-4ce3-8132-b00da958bd19" (UID: "9a169dae-6fcb-4ce3-8132-b00da958bd19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:41.788321 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:41.788290    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-xks-ca" pod "9a169dae-6fcb-4ce3-8132-b00da958bd19" (UID: "9a169dae-6fcb-4ce3-8132-b00da958bd19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:41.789480 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:41.789451    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a169dae-6fcb-4ce3-8132-b00da958bd19-kube-api-access-v9g6j" pod "9a169dae-6fcb-4ce3-8132-b00da958bd19" (UID: "9a169dae-6fcb-4ce3-8132-b00da958bd19"). InnerVolumeSpecName "kube-api-access-v9g6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:41.899085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:41.890763    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9g6j\" (UniqueName: \"kubernetes.io/projected/9a169dae-6fcb-4ce3-8132-b00da958bd19-kube-api-access-v9g6j\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:41.899085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:41.890798    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:41.899085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:41.890814    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a169dae-6fcb-4ce3-8132-b00da958bd19-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:42.592664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:42.591514    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx" event={"ID":"9a169dae-6fcb-4ce3-8132-b00da958bd19","Type":"ContainerDied","Data":"2c1b061c97e3551120c04afffdb14931898a0d85fb115a18319e85ddecd63b72"}
Oct 29 22:13:42.592664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:42.591577    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c1b061c97e3551120c04afffdb14931898a0d85fb115a18319e85ddecd63b72"
Oct 29 22:13:42.592664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:42.591673    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"
Oct 29 22:13:44.480265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:44.480229    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"]
Oct 29 22:13:44.480739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:44.480546    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy" containerID="cri-o://7b4453e6564e1d172be133b09ec124a1627a6e6b6744dfa4a360197701c78b77" gracePeriod=30
Oct 29 22:13:44.491103 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:44.491077    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"]
Oct 29 22:13:44.491681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:44.491575    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg" podUID="9e3763e5-e3ad-4c90-b6c8-1581e3f45788" containerName="istio-proxy" containerID="cri-o://c046596fccd88a7f342eba164c1b75fc39643f1e38edcaefb35a68539680387a" gracePeriod=30
Oct 29 22:13:44.537153 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:44.537119    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"]
Oct 29 22:13:44.537679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:44.537611    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h" podUID="7650af16-12ae-49f4-93c0-9340911d29ab" containerName="istio-proxy" containerID="cri-o://1f675d5008e6cde64af9cde6a573a68a05494795bc074c35fd834cc4624fed70" gracePeriod=30
Oct 29 22:13:45.165285 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.165241    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"]
Oct 29 22:13:45.169209 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.169185    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4e8583-sqx8h"]
Oct 29 22:13:45.344651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.341039    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"]
Oct 29 22:13:45.344651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.342192    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9a169dae-6fcb-4ce3-8132-b00da958bd19" containerName="revoke-keys"
Oct 29 22:13:45.344651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.342214    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a169dae-6fcb-4ce3-8132-b00da958bd19" containerName="revoke-keys"
Oct 29 22:13:45.344651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.342482    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a169dae-6fcb-4ce3-8132-b00da958bd19" containerName="revoke-keys"
Oct 29 22:13:45.380215 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.380177    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"]
Oct 29 22:13:45.380572 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.380551    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"
Oct 29 22:13:45.401912 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.401880    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:13:45.427436 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.402349    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:13:45.427695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.403070    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:13:45.427853 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.403133    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:13:45.540102 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.540047    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p\" (UID: \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"
Oct 29 22:13:45.540477 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.540225    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj7l6\" (UniqueName: \"kubernetes.io/projected/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-kube-api-access-zj7l6\") pod \"maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p\" (UID: \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"
Oct 29 22:13:45.540477 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.540302    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p\" (UID: \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"
Oct 29 22:13:45.632771 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.632740    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a527702-a704-4c3e-8e19-acf1d5bb9677" path="/var/lib/kubelet/pods/9a527702-a704-4c3e-8e19-acf1d5bb9677/volumes"
Oct 29 22:13:45.641784 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.641755    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj7l6\" (UniqueName: \"kubernetes.io/projected/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-kube-api-access-zj7l6\") pod \"maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p\" (UID: \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"
Oct 29 22:13:45.641939 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.641815    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p\" (UID: \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"
Oct 29 22:13:45.642095 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.642071    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p\" (UID: \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"
Oct 29 22:13:45.642546 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.642523    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p\" (UID: \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"
Oct 29 22:13:45.644674 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.644606    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p\" (UID: \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"
Oct 29 22:13:45.664255 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.664232    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj7l6\" (UniqueName: \"kubernetes.io/projected/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-kube-api-access-zj7l6\") pod \"maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p\" (UID: \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"
Oct 29 22:13:45.755485 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:45.755416    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"
Oct 29 22:13:46.070670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:46.070592    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"]
Oct 29 22:13:46.072452 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:13:46.072428    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8065e1cd_dfe3_4a1e_83f0_58d970f03ff9.slice/crio-979d0622b62309970ab0f11bd88581955929ef318ac19dd45eae229bf9dffae1 WatchSource:0}: Error finding container 979d0622b62309970ab0f11bd88581955929ef318ac19dd45eae229bf9dffae1: Status 404 returned error can't find the container with id 979d0622b62309970ab0f11bd88581955929ef318ac19dd45eae229bf9dffae1
Oct 29 22:13:46.615417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:46.615370    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2" podUID="efe88fdd-8cf4-4758-8d81-546d50c2786f" containerName="payload-processing" containerID="cri-o://34ff6b30ffbac0f10e1d8a832bc405c73f1a8c4ac90c59517c183b0cbe080f95" gracePeriod=25
Oct 29 22:13:46.632375 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:46.632344    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p" event={"ID":"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9","Type":"ContainerStarted","Data":"a31a0815ed3234d326845e0b6ca670fc71fd390a95f49dffda9bc41726220a9c"}
Oct 29 22:13:46.632521 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:46.632396    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p" event={"ID":"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9","Type":"ContainerStarted","Data":"979d0622b62309970ab0f11bd88581955929ef318ac19dd45eae229bf9dffae1"}
Oct 29 22:13:46.646225 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:46.646150    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx" podUID="0369fe15-bc84-442e-973c-fde55e95785a" containerName="payload-pre-processing" containerID="cri-o://6739c9b6994f6d5bc303ce2ed054bc240cb72d5c53445eb5677cd69e878a61f5" gracePeriod=25
Oct 29 22:13:46.676490 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:46.676427    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p" podStartSLOduration=1.676410277 podStartE2EDuration="1.676410277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:46.671961721 +0000 UTC m=+1599.172004019" watchObservedRunningTime="2026-10-29 22:13:46.676410277 +0000 UTC m=+1599.176452580"
Oct 29 22:13:47.187534 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:47.187498    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:13:47.187723 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:47.187564    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:13:47.762821 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:47.762781    7082 generic.go:356] "Generic (PLEG): container finished" podID="8065e1cd-dfe3-4a1e-83f0-58d970f03ff9" containerID="a31a0815ed3234d326845e0b6ca670fc71fd390a95f49dffda9bc41726220a9c" exitCode=0
Oct 29 22:13:47.781110 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:47.765254    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p" event={"ID":"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9","Type":"ContainerDied","Data":"a31a0815ed3234d326845e0b6ca670fc71fd390a95f49dffda9bc41726220a9c"}
Oct 29 22:13:47.958142 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:47.958108    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:13:47.958313 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:47.958165    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg" podUID="9e3763e5-e3ad-4c90-b6c8-1581e3f45788" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:13:48.031115 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:48.031044    7082 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:13:48.031265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:48.031114    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h" podUID="7650af16-12ae-49f4-93c0-9340911d29ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:13:48.969142 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:48.969106    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"
Oct 29 22:13:49.000827 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.000799    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"]
Oct 29 22:13:49.001711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.001689    7082 cpu_manager.go:414] "containerMap: removing container" podUID="8065e1cd-dfe3-4a1e-83f0-58d970f03ff9" containerName="revoke-keys"
Oct 29 22:13:49.001711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.001709    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8065e1cd-dfe3-4a1e-83f0-58d970f03ff9" containerName="revoke-keys"
Oct 29 22:13:49.001910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.001897    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="8065e1cd-dfe3-4a1e-83f0-58d970f03ff9" containerName="revoke-keys"
Oct 29 22:13:49.005965 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.005946    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"
Oct 29 22:13:49.027849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.027825    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"]
Oct 29 22:13:49.049801 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.049738    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"]
Oct 29 22:13:49.050022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.050000    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"
Oct 29 22:13:49.050132 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.050016    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"]
Oct 29 22:13:49.063124 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.063096    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-kube-api-access-zj7l6\" (UniqueName: \"kubernetes.io/projected/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-kube-api-access-zj7l6\") pod \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\" (UID: \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\") "
Oct 29 22:13:49.063238 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.063137    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-xks-ca\") pod \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\" (UID: \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\") "
Oct 29 22:13:49.063320 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.063293    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-openshift-ca\") pod \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\" (UID: \"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9\") "
Oct 29 22:13:49.064304 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.064282    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-openshift-ca" pod "8065e1cd-dfe3-4a1e-83f0-58d970f03ff9" (UID: "8065e1cd-dfe3-4a1e-83f0-58d970f03ff9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:49.066053 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.066032    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-kube-api-access-zj7l6" pod "8065e1cd-dfe3-4a1e-83f0-58d970f03ff9" (UID: "8065e1cd-dfe3-4a1e-83f0-58d970f03ff9"). InnerVolumeSpecName "kube-api-access-zj7l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:49.068536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.068508    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-xks-ca" pod "8065e1cd-dfe3-4a1e-83f0-58d970f03ff9" (UID: "8065e1cd-dfe3-4a1e-83f0-58d970f03ff9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:49.165424 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.165385    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ae5fb64-5vnxs\" (UID: \"b720e32b-c20f-48a1-ac2a-b26405ef141f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"
Oct 29 22:13:49.165565 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.165440    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eedc34b-fzflz\" (UID: \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"
Oct 29 22:13:49.165565 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.165524    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwxdc\" (UniqueName: \"kubernetes.io/projected/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-kube-api-access-cwxdc\") pod \"maas-api-revoke-sub-3eedc34b-fzflz\" (UID: \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"
Oct 29 22:13:49.165684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.165588    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eedc34b-fzflz\" (UID: \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"
Oct 29 22:13:49.165762 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.165740    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ae5fb64-5vnxs\" (UID: \"b720e32b-c20f-48a1-ac2a-b26405ef141f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"
Oct 29 22:13:49.165863 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.165851    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j4z4\" (UniqueName: \"kubernetes.io/projected/b720e32b-c20f-48a1-ac2a-b26405ef141f-kube-api-access-5j4z4\") pod \"maas-api-revoke-sub-4ae5fb64-5vnxs\" (UID: \"b720e32b-c20f-48a1-ac2a-b26405ef141f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"
Oct 29 22:13:49.166060 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.166049    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj7l6\" (UniqueName: \"kubernetes.io/projected/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-kube-api-access-zj7l6\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:49.166098 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.166065    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:49.166098 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.166081    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:49.267423 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.267393    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ae5fb64-5vnxs\" (UID: \"b720e32b-c20f-48a1-ac2a-b26405ef141f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"
Oct 29 22:13:49.267576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.267479    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j4z4\" (UniqueName: \"kubernetes.io/projected/b720e32b-c20f-48a1-ac2a-b26405ef141f-kube-api-access-5j4z4\") pod \"maas-api-revoke-sub-4ae5fb64-5vnxs\" (UID: \"b720e32b-c20f-48a1-ac2a-b26405ef141f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"
Oct 29 22:13:49.267693 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.267677    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ae5fb64-5vnxs\" (UID: \"b720e32b-c20f-48a1-ac2a-b26405ef141f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"
Oct 29 22:13:49.267753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.267714    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eedc34b-fzflz\" (UID: \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"
Oct 29 22:13:49.267810 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.267759    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwxdc\" (UniqueName: \"kubernetes.io/projected/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-kube-api-access-cwxdc\") pod \"maas-api-revoke-sub-3eedc34b-fzflz\" (UID: \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"
Oct 29 22:13:49.267861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.267831    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eedc34b-fzflz\" (UID: \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"
Oct 29 22:13:49.268279 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.268246    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ae5fb64-5vnxs\" (UID: \"b720e32b-c20f-48a1-ac2a-b26405ef141f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"
Oct 29 22:13:49.268514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.268468    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eedc34b-fzflz\" (UID: \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"
Oct 29 22:13:49.270280 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.270251    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eedc34b-fzflz\" (UID: \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"
Oct 29 22:13:49.270392 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.270326    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ae5fb64-5vnxs\" (UID: \"b720e32b-c20f-48a1-ac2a-b26405ef141f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"
Oct 29 22:13:49.290549 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.290521    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j4z4\" (UniqueName: \"kubernetes.io/projected/b720e32b-c20f-48a1-ac2a-b26405ef141f-kube-api-access-5j4z4\") pod \"maas-api-revoke-sub-4ae5fb64-5vnxs\" (UID: \"b720e32b-c20f-48a1-ac2a-b26405ef141f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"
Oct 29 22:13:49.290692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.290599    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwxdc\" (UniqueName: \"kubernetes.io/projected/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-kube-api-access-cwxdc\") pod \"maas-api-revoke-sub-3eedc34b-fzflz\" (UID: \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"
Oct 29 22:13:49.349299 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.349273    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"
Oct 29 22:13:49.372843 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.372814    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"
Oct 29 22:13:49.808532 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.808501    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"]
Oct 29 22:13:49.809201 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:13:49.809125    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb720e32b_c20f_48a1_ac2a_b26405ef141f.slice/crio-4adb4f998b905a00e2c1b53bb8da2505a28ad63779e0869797d76c59427dd65c WatchSource:0}: Error finding container 4adb4f998b905a00e2c1b53bb8da2505a28ad63779e0869797d76c59427dd65c: Status 404 returned error can't find the container with id 4adb4f998b905a00e2c1b53bb8da2505a28ad63779e0869797d76c59427dd65c
Oct 29 22:13:49.821776 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.821744    7082 generic.go:356] "Generic (PLEG): container finished" podID="9e3763e5-e3ad-4c90-b6c8-1581e3f45788" containerID="c046596fccd88a7f342eba164c1b75fc39643f1e38edcaefb35a68539680387a" exitCode=0
Oct 29 22:13:49.821887 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.821841    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg" event={"ID":"9e3763e5-e3ad-4c90-b6c8-1581e3f45788","Type":"ContainerDied","Data":"c046596fccd88a7f342eba164c1b75fc39643f1e38edcaefb35a68539680387a"}
Oct 29 22:13:49.824795 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.824767    7082 generic.go:356] "Generic (PLEG): container finished" podID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerID="7b4453e6564e1d172be133b09ec124a1627a6e6b6744dfa4a360197701c78b77" exitCode=0
Oct 29 22:13:49.825317 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.824937    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" event={"ID":"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6","Type":"ContainerDied","Data":"7b4453e6564e1d172be133b09ec124a1627a6e6b6744dfa4a360197701c78b77"}
Oct 29 22:13:49.825576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.825561    7082 scope.go:122] "RemoveContainer" containerID="3d32fa4d317e07e36677770998a56d106e1161ca3a787fb645f99f769272193e"
Oct 29 22:13:49.827947 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.827847    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p" event={"ID":"8065e1cd-dfe3-4a1e-83f0-58d970f03ff9","Type":"ContainerDied","Data":"979d0622b62309970ab0f11bd88581955929ef318ac19dd45eae229bf9dffae1"}
Oct 29 22:13:49.827947 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.827889    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="979d0622b62309970ab0f11bd88581955929ef318ac19dd45eae229bf9dffae1"
Oct 29 22:13:49.828181 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.827952    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"
Oct 29 22:13:49.831937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.831918    7082 generic.go:356] "Generic (PLEG): container finished" podID="7650af16-12ae-49f4-93c0-9340911d29ab" containerID="1f675d5008e6cde64af9cde6a573a68a05494795bc074c35fd834cc4624fed70" exitCode=0
Oct 29 22:13:49.832089 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.832072    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h" event={"ID":"7650af16-12ae-49f4-93c0-9340911d29ab","Type":"ContainerDied","Data":"1f675d5008e6cde64af9cde6a573a68a05494795bc074c35fd834cc4624fed70"}
Oct 29 22:13:49.888676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.885735    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"]
Oct 29 22:13:49.888676 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:13:49.887341    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03cd98c0_0f31_4b7d_8c0d_c15e827315c8.slice/crio-7315c5db66b02d32664eb4c547502c9cd1d066479d0489b21a42e89a1b4814f1 WatchSource:0}: Error finding container 7315c5db66b02d32664eb4c547502c9cd1d066479d0489b21a42e89a1b4814f1: Status 404 returned error can't find the container with id 7315c5db66b02d32664eb4c547502c9cd1d066479d0489b21a42e89a1b4814f1
Oct 29 22:13:49.892407 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.892374    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:13:49.930418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.928942    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istiod-ca-cert\") pod \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") "
Oct 29 22:13:49.930418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.928992    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-data\") pod \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") "
Oct 29 22:13:49.930418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.929030    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-certs\") pod \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") "
Oct 29 22:13:49.930418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.929086    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-podinfo\") pod \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") "
Oct 29 22:13:49.930418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.929148    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-token\" (UniqueName: \"kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-token\") pod \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") "
Oct 29 22:13:49.930418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.929183    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-envoy\") pod \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") "
Oct 29 22:13:49.930418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.929243    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-kube-api-access-sg4dd\" (UniqueName: \"kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-kube-api-access-sg4dd\") pod \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") "
Oct 29 22:13:49.930418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.929325    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-socket\") pod \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") "
Oct 29 22:13:49.930418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.929571    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-credential-socket\") pod \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\" (UID: \"9e3763e5-e3ad-4c90-b6c8-1581e3f45788\") "
Oct 29 22:13:49.931442 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.931112    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-credential-socket" pod "9e3763e5-e3ad-4c90-b6c8-1581e3f45788" (UID: "9e3763e5-e3ad-4c90-b6c8-1581e3f45788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.933071 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.933004    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-certs" pod "9e3763e5-e3ad-4c90-b6c8-1581e3f45788" (UID: "9e3763e5-e3ad-4c90-b6c8-1581e3f45788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.933071 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.933049    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istiod-ca-cert" pod "9e3763e5-e3ad-4c90-b6c8-1581e3f45788" (UID: "9e3763e5-e3ad-4c90-b6c8-1581e3f45788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:49.933703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.933668    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-socket" pod "9e3763e5-e3ad-4c90-b6c8-1581e3f45788" (UID: "9e3763e5-e3ad-4c90-b6c8-1581e3f45788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.934016 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.933998    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-data" pod "9e3763e5-e3ad-4c90-b6c8-1581e3f45788" (UID: "9e3763e5-e3ad-4c90-b6c8-1581e3f45788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.986110 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:49.978855    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"]
Oct 29 22:13:50.005243 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.005078    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-podinfo" pod "9e3763e5-e3ad-4c90-b6c8-1581e3f45788" (UID: "9e3763e5-e3ad-4c90-b6c8-1581e3f45788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:50.014500 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.013832    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-envoy" pod "9e3763e5-e3ad-4c90-b6c8-1581e3f45788" (UID: "9e3763e5-e3ad-4c90-b6c8-1581e3f45788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.014500 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.013895    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-kube-api-access-sg4dd" pod "9e3763e5-e3ad-4c90-b6c8-1581e3f45788" (UID: "9e3763e5-e3ad-4c90-b6c8-1581e3f45788"). InnerVolumeSpecName "kube-api-access-sg4dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.014500 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.013930    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-token" pod "9e3763e5-e3ad-4c90-b6c8-1581e3f45788" (UID: "9e3763e5-e3ad-4c90-b6c8-1581e3f45788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.038248 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.038205    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.038248 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.038242    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.038437 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.038257    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.038437 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.038269    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.038437 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.038281    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.038437 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.038293    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.038437 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.038306    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.038437 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.038319    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg4dd\" (UniqueName: \"kubernetes.io/projected/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-kube-api-access-sg4dd\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.038437 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.038333    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3763e5-e3ad-4c90-b6c8-1581e3f45788-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.049506 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.049484    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"]
Oct 29 22:13:50.146549 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.146523    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:13:50.151169 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.151146    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:13:50.247281 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247248    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-envoy\") pod \"7650af16-12ae-49f4-93c0-9340911d29ab\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") "
Oct 29 22:13:50.247466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247293    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-podinfo\") pod \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") "
Oct 29 22:13:50.247466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247341    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-kube-api-access-bfbt8\" (UniqueName: \"kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-kube-api-access-bfbt8\") pod \"7650af16-12ae-49f4-93c0-9340911d29ab\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") "
Oct 29 22:13:50.247466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247368    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7650af16-12ae-49f4-93c0-9340911d29ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7650af16-12ae-49f4-93c0-9340911d29ab-istiod-ca-cert\") pod \"7650af16-12ae-49f4-93c0-9340911d29ab\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") "
Oct 29 22:13:50.247466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247395    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-socket\") pod \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") "
Oct 29 22:13:50.247466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247423    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-credential-socket\") pod \"7650af16-12ae-49f4-93c0-9340911d29ab\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") "
Oct 29 22:13:50.247907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247481    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-credential-socket\") pod \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") "
Oct 29 22:13:50.247907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247505    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-data\") pod \"7650af16-12ae-49f4-93c0-9340911d29ab\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") "
Oct 29 22:13:50.247907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247562    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-certs\") pod \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") "
Oct 29 22:13:50.247907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247614    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-certs\") pod \"7650af16-12ae-49f4-93c0-9340911d29ab\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") "
Oct 29 22:13:50.247907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247669    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-kube-api-access-d79nx\" (UniqueName: \"kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-kube-api-access-d79nx\") pod \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") "
Oct 29 22:13:50.247907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247693    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-envoy\") pod \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") "
Oct 29 22:13:50.247907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247716    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-data\") pod \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") "
Oct 29 22:13:50.247907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247838    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istiod-ca-cert\") pod \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") "
Oct 29 22:13:50.247907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247899    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-istio-token\" (UniqueName: \"kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-istio-token\") pod \"7650af16-12ae-49f4-93c0-9340911d29ab\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") "
Oct 29 22:13:50.248403 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247942    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-token\" (UniqueName: \"kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-token\") pod \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\" (UID: \"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6\") "
Oct 29 22:13:50.248403 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247962    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7650af16-12ae-49f4-93c0-9340911d29ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7650af16-12ae-49f4-93c0-9340911d29ab-istio-podinfo\") pod \"7650af16-12ae-49f4-93c0-9340911d29ab\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") "
Oct 29 22:13:50.248403 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.247993    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-socket\") pod \"7650af16-12ae-49f4-93c0-9340911d29ab\" (UID: \"7650af16-12ae-49f4-93c0-9340911d29ab\") "
Oct 29 22:13:50.248803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.248783    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-socket" pod "7650af16-12ae-49f4-93c0-9340911d29ab" (UID: "7650af16-12ae-49f4-93c0-9340911d29ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.248936 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.248904    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-certs" pod "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" (UID: "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.249124 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.249107    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-certs" pod "7650af16-12ae-49f4-93c0-9340911d29ab" (UID: "7650af16-12ae-49f4-93c0-9340911d29ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.249514 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.249488    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.249617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.249520    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.249617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.249534    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.249966 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.249779    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-socket" pod "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" (UID: "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.251009 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.250753    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-data" pod "7650af16-12ae-49f4-93c0-9340911d29ab" (UID: "7650af16-12ae-49f4-93c0-9340911d29ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.251341 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.251145    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-data" pod "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" (UID: "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.252671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.251478    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-credential-socket" pod "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" (UID: "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.252761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.252708    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-credential-socket" pod "7650af16-12ae-49f4-93c0-9340911d29ab" (UID: "7650af16-12ae-49f4-93c0-9340911d29ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.253503 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.253208    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7650af16-12ae-49f4-93c0-9340911d29ab-istiod-ca-cert" pod "7650af16-12ae-49f4-93c0-9340911d29ab" (UID: "7650af16-12ae-49f4-93c0-9340911d29ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:50.253503 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.253265    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istiod-ca-cert" pod "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" (UID: "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:50.289140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.288895    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-envoy" pod "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" (UID: "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.289140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.288951    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-kube-api-access-bfbt8" pod "7650af16-12ae-49f4-93c0-9340911d29ab" (UID: "7650af16-12ae-49f4-93c0-9340911d29ab"). InnerVolumeSpecName "kube-api-access-bfbt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.289140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.289007    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-kube-api-access-d79nx" pod "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" (UID: "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6"). InnerVolumeSpecName "kube-api-access-d79nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.289140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.289025    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-istio-token" pod "7650af16-12ae-49f4-93c0-9340911d29ab" (UID: "7650af16-12ae-49f4-93c0-9340911d29ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.289140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.289063    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-podinfo" pod "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" (UID: "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:50.289140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.289078    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-token" pod "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" (UID: "2702b7ee-1ae0-4831-92d4-ae6d7bb408c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.289140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.289112    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-envoy" pod "7650af16-12ae-49f4-93c0-9340911d29ab" (UID: "7650af16-12ae-49f4-93c0-9340911d29ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.293248 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.293209    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7650af16-12ae-49f4-93c0-9340911d29ab-istio-podinfo" pod "7650af16-12ae-49f4-93c0-9340911d29ab" (UID: "7650af16-12ae-49f4-93c0-9340911d29ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:50.351181 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351156    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfbt8\" (UniqueName: \"kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-kube-api-access-bfbt8\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351181 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351181    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7650af16-12ae-49f4-93c0-9340911d29ab-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351191    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351202    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351211    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351219    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351227    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d79nx\" (UniqueName: \"kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-kube-api-access-d79nx\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351236    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351245    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351254    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351261    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7650af16-12ae-49f4-93c0-9340911d29ab-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351268    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351275    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7650af16-12ae-49f4-93c0-9340911d29ab-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351282    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7650af16-12ae-49f4-93c0-9340911d29ab-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.351352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.351289    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.842279 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.842255    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"
Oct 29 22:13:50.842409 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.842256    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg" event={"ID":"9e3763e5-e3ad-4c90-b6c8-1581e3f45788","Type":"ContainerDied","Data":"a432dba38394a461ce52a7b7422dca14351e30d8bf47ebcbeaadbb9774a4da86"}
Oct 29 22:13:50.842409 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.842389    7082 scope.go:122] "RemoveContainer" containerID="c046596fccd88a7f342eba164c1b75fc39643f1e38edcaefb35a68539680387a"
Oct 29 22:13:50.843831 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.843804    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"
Oct 29 22:13:50.843941 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.843835    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68" event={"ID":"2702b7ee-1ae0-4831-92d4-ae6d7bb408c6","Type":"ContainerDied","Data":"657ce4a2b69cc9f32498cd6d11a4c65745b96075788cd21628db2e66c860e168"}
Oct 29 22:13:50.845976 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.845951    7082 generic.go:356] "Generic (PLEG): container finished" podID="b720e32b-c20f-48a1-ac2a-b26405ef141f" containerID="135157cc58611b152b1765ed399e954c4eb97534c6ff3054305563d0d7c5506d" exitCode=0
Oct 29 22:13:50.846077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.846055    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs" event={"ID":"b720e32b-c20f-48a1-ac2a-b26405ef141f","Type":"ContainerDied","Data":"135157cc58611b152b1765ed399e954c4eb97534c6ff3054305563d0d7c5506d"}
Oct 29 22:13:50.846143 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.846126    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs" event={"ID":"b720e32b-c20f-48a1-ac2a-b26405ef141f","Type":"ContainerStarted","Data":"4adb4f998b905a00e2c1b53bb8da2505a28ad63779e0869797d76c59427dd65c"}
Oct 29 22:13:50.849060 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.849041    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"
Oct 29 22:13:50.849159 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.849076    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h" event={"ID":"7650af16-12ae-49f4-93c0-9340911d29ab","Type":"ContainerDied","Data":"dd63eb718ef23f1e22b24e923858d08a623b86b30943f674c28a31b2df7d5329"}
Oct 29 22:13:50.850989 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.850970    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz" event={"ID":"03cd98c0-0f31-4b7d-8c0d-c15e827315c8","Type":"ContainerStarted","Data":"21b0233ab7e93107a4b9e4a5dee8dcd908f6853f32e631a6fb8c151cdf28cfcc"}
Oct 29 22:13:50.851062 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.851008    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz" event={"ID":"03cd98c0-0f31-4b7d-8c0d-c15e827315c8","Type":"ContainerStarted","Data":"7315c5db66b02d32664eb4c547502c9cd1d066479d0489b21a42e89a1b4814f1"}
Oct 29 22:13:50.872786 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.872762    7082 scope.go:122] "RemoveContainer" containerID="7b4453e6564e1d172be133b09ec124a1627a6e6b6744dfa4a360197701c78b77"
Oct 29 22:13:50.883530 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.883513    7082 scope.go:122] "RemoveContainer" containerID="1f675d5008e6cde64af9cde6a573a68a05494795bc074c35fd834cc4624fed70"
Oct 29 22:13:50.986557 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:50.986504    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz" podStartSLOduration=2.986490703 podStartE2EDuration="2.986490703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:50.983556186 +0000 UTC m=+1603.483598486" watchObservedRunningTime="2026-10-29 22:13:50.986490703 +0000 UTC m=+1603.486533002"
Oct 29 22:13:51.038830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:51.038798    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"]
Oct 29 22:13:51.043701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:51.043676    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bnw5h"]
Oct 29 22:13:51.074563 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:51.074509    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"]
Oct 29 22:13:51.078155 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:51.078134    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-dmfjg"]
Oct 29 22:13:51.118698 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:51.118671    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"]
Oct 29 22:13:51.122082 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:51.122062    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-bkr68"]
Oct 29 22:13:51.634515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:51.634483    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" path="/var/lib/kubelet/pods/2702b7ee-1ae0-4831-92d4-ae6d7bb408c6/volumes"
Oct 29 22:13:51.635194 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:51.635173    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7650af16-12ae-49f4-93c0-9340911d29ab" path="/var/lib/kubelet/pods/7650af16-12ae-49f4-93c0-9340911d29ab/volumes"
Oct 29 22:13:51.635811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:51.635794    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3763e5-e3ad-4c90-b6c8-1581e3f45788" path="/var/lib/kubelet/pods/9e3763e5-e3ad-4c90-b6c8-1581e3f45788/volumes"
Oct 29 22:13:51.866537 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:51.866501    7082 generic.go:356] "Generic (PLEG): container finished" podID="03cd98c0-0f31-4b7d-8c0d-c15e827315c8" containerID="21b0233ab7e93107a4b9e4a5dee8dcd908f6853f32e631a6fb8c151cdf28cfcc" exitCode=0
Oct 29 22:13:51.866772 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:51.866539    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz" event={"ID":"03cd98c0-0f31-4b7d-8c0d-c15e827315c8","Type":"ContainerDied","Data":"21b0233ab7e93107a4b9e4a5dee8dcd908f6853f32e631a6fb8c151cdf28cfcc"}
Oct 29 22:13:52.085109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.085078    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"
Oct 29 22:13:52.240203 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.240124    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b720e32b-c20f-48a1-ac2a-b26405ef141f-kube-api-access-5j4z4\" (UniqueName: \"kubernetes.io/projected/b720e32b-c20f-48a1-ac2a-b26405ef141f-kube-api-access-5j4z4\") pod \"b720e32b-c20f-48a1-ac2a-b26405ef141f\" (UID: \"b720e32b-c20f-48a1-ac2a-b26405ef141f\") "
Oct 29 22:13:52.240203 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.240194    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-openshift-ca\") pod \"b720e32b-c20f-48a1-ac2a-b26405ef141f\" (UID: \"b720e32b-c20f-48a1-ac2a-b26405ef141f\") "
Oct 29 22:13:52.240402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.240379    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-xks-ca\") pod \"b720e32b-c20f-48a1-ac2a-b26405ef141f\" (UID: \"b720e32b-c20f-48a1-ac2a-b26405ef141f\") "
Oct 29 22:13:52.240561 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.240537    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-openshift-ca" pod "b720e32b-c20f-48a1-ac2a-b26405ef141f" (UID: "b720e32b-c20f-48a1-ac2a-b26405ef141f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:52.241104 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.241036    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.242582 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.242556    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-xks-ca" pod "b720e32b-c20f-48a1-ac2a-b26405ef141f" (UID: "b720e32b-c20f-48a1-ac2a-b26405ef141f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:52.242706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.242647    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b720e32b-c20f-48a1-ac2a-b26405ef141f-kube-api-access-5j4z4" pod "b720e32b-c20f-48a1-ac2a-b26405ef141f" (UID: "b720e32b-c20f-48a1-ac2a-b26405ef141f"). InnerVolumeSpecName "kube-api-access-5j4z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:52.342007 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.341978    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j4z4\" (UniqueName: \"kubernetes.io/projected/b720e32b-c20f-48a1-ac2a-b26405ef141f-kube-api-access-5j4z4\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.342007 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.342006    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b720e32b-c20f-48a1-ac2a-b26405ef141f-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.830257 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.830219    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx" podUID="0369fe15-bc84-442e-973c-fde55e95785a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:13:52.876774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.876744    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs"
Oct 29 22:13:52.876928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.876783    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ae5fb64-5vnxs" event={"ID":"b720e32b-c20f-48a1-ac2a-b26405ef141f","Type":"ContainerDied","Data":"4adb4f998b905a00e2c1b53bb8da2505a28ad63779e0869797d76c59427dd65c"}
Oct 29 22:13:52.876928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:52.876832    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4adb4f998b905a00e2c1b53bb8da2505a28ad63779e0869797d76c59427dd65c"
Oct 29 22:13:53.257855 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.257821    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"
Oct 29 22:13:53.379188 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.379112    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-openshift-ca\") pod \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\" (UID: \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\") "
Oct 29 22:13:53.379326 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.379264    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-xks-ca\") pod \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\" (UID: \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\") "
Oct 29 22:13:53.379447 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.379427    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-kube-api-access-cwxdc\" (UniqueName: \"kubernetes.io/projected/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-kube-api-access-cwxdc\") pod \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\" (UID: \"03cd98c0-0f31-4b7d-8c0d-c15e827315c8\") "
Oct 29 22:13:53.379562 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.379428    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-openshift-ca" pod "03cd98c0-0f31-4b7d-8c0d-c15e827315c8" (UID: "03cd98c0-0f31-4b7d-8c0d-c15e827315c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:53.380322 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.380294    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:53.381479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.381452    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-xks-ca" pod "03cd98c0-0f31-4b7d-8c0d-c15e827315c8" (UID: "03cd98c0-0f31-4b7d-8c0d-c15e827315c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:53.381589 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.381541    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-kube-api-access-cwxdc" pod "03cd98c0-0f31-4b7d-8c0d-c15e827315c8" (UID: "03cd98c0-0f31-4b7d-8c0d-c15e827315c8"). InnerVolumeSpecName "kube-api-access-cwxdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:53.497652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.482081    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwxdc\" (UniqueName: \"kubernetes.io/projected/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-kube-api-access-cwxdc\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:53.536655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.520704    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03cd98c0-0f31-4b7d-8c0d-c15e827315c8-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:53.839309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.839270    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2" podUID="efe88fdd-8cf4-4758-8d81-546d50c2786f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:13:53.887269 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.887238    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz"
Oct 29 22:13:53.887269 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.887245    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eedc34b-fzflz" event={"ID":"03cd98c0-0f31-4b7d-8c0d-c15e827315c8","Type":"ContainerDied","Data":"7315c5db66b02d32664eb4c547502c9cd1d066479d0489b21a42e89a1b4814f1"}
Oct 29 22:13:53.887477 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:53.887296    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7315c5db66b02d32664eb4c547502c9cd1d066479d0489b21a42e89a1b4814f1"
Oct 29 22:13:54.022100 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.022060    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"]
Oct 29 22:13:54.022821 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.022802    7082 cpu_manager.go:414] "containerMap: removing container" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy"
Oct 29 22:13:54.022821 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.022820    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy"
Oct 29 22:13:54.022984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.022837    7082 cpu_manager.go:414] "containerMap: removing container" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy"
Oct 29 22:13:54.022984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.022843    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy"
Oct 29 22:13:54.022984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.022860    7082 cpu_manager.go:414] "containerMap: removing container" podUID="03cd98c0-0f31-4b7d-8c0d-c15e827315c8" containerName="revoke-keys"
Oct 29 22:13:54.022984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.022865    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03cd98c0-0f31-4b7d-8c0d-c15e827315c8" containerName="revoke-keys"
Oct 29 22:13:54.022984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.022876    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9e3763e5-e3ad-4c90-b6c8-1581e3f45788" containerName="istio-proxy"
Oct 29 22:13:54.022984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.022881    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3763e5-e3ad-4c90-b6c8-1581e3f45788" containerName="istio-proxy"
Oct 29 22:13:54.022984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.022904    7082 cpu_manager.go:414] "containerMap: removing container" podUID="b720e32b-c20f-48a1-ac2a-b26405ef141f" containerName="revoke-keys"
Oct 29 22:13:54.022984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.022909    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b720e32b-c20f-48a1-ac2a-b26405ef141f" containerName="revoke-keys"
Oct 29 22:13:54.022984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.022915    7082 cpu_manager.go:414] "containerMap: removing container" podUID="7650af16-12ae-49f4-93c0-9340911d29ab" containerName="istio-proxy"
Oct 29 22:13:54.022984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.022919    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7650af16-12ae-49f4-93c0-9340911d29ab" containerName="istio-proxy"
Oct 29 22:13:54.023500 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.023019    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy"
Oct 29 22:13:54.023500 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.023029    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="03cd98c0-0f31-4b7d-8c0d-c15e827315c8" containerName="revoke-keys"
Oct 29 22:13:54.023500 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.023035    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3763e5-e3ad-4c90-b6c8-1581e3f45788" containerName="istio-proxy"
Oct 29 22:13:54.023500 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.023046    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="b720e32b-c20f-48a1-ac2a-b26405ef141f" containerName="revoke-keys"
Oct 29 22:13:54.023500 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.023053    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="7650af16-12ae-49f4-93c0-9340911d29ab" containerName="istio-proxy"
Oct 29 22:13:54.027084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.027061    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"
Oct 29 22:13:54.042950 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.042926    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:13:54.043152 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.043033    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:13:54.043152 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.043096    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:13:54.043152 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.042925    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:13:54.048348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.048324    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"]
Oct 29 22:13:54.166474 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.166439    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4\" (UID: \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"
Oct 29 22:13:54.166621 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.166489    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4\" (UID: \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"
Oct 29 22:13:54.166751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.166727    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd7pq\" (UniqueName: \"kubernetes.io/projected/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-kube-api-access-zd7pq\") pod \"maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4\" (UID: \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"
Oct 29 22:13:54.268686 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.268658    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd7pq\" (UniqueName: \"kubernetes.io/projected/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-kube-api-access-zd7pq\") pod \"maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4\" (UID: \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"
Oct 29 22:13:54.269100 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.268839    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4\" (UID: \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"
Oct 29 22:13:54.269100 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.268871    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4\" (UID: \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"
Oct 29 22:13:54.269523 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.269494    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4\" (UID: \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"
Oct 29 22:13:54.271329 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.271288    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4\" (UID: \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"
Oct 29 22:13:54.291759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.291706    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd7pq\" (UniqueName: \"kubernetes.io/projected/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-kube-api-access-zd7pq\") pod \"maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4\" (UID: \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"
Oct 29 22:13:54.344560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.344532    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"
Oct 29 22:13:54.665425 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.665394    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"]
Oct 29 22:13:54.666161 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:13:54.666130    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20783eec_14f3_4dd3_ada8_cd52fcbe6f98.slice/crio-bf9d89490637821f0662f9d44d25711a373853b3c47994e2a69424b6584a6049 WatchSource:0}: Error finding container bf9d89490637821f0662f9d44d25711a373853b3c47994e2a69424b6584a6049: Status 404 returned error can't find the container with id bf9d89490637821f0662f9d44d25711a373853b3c47994e2a69424b6584a6049
Oct 29 22:13:54.899450 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:54.899417    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4" event={"ID":"20783eec-14f3-4dd3-ada8-cd52fcbe6f98","Type":"ContainerStarted","Data":"bf9d89490637821f0662f9d44d25711a373853b3c47994e2a69424b6584a6049"}
Oct 29 22:13:55.911635 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:55.911594    7082 generic.go:356] "Generic (PLEG): container finished" podID="20783eec-14f3-4dd3-ada8-cd52fcbe6f98" containerID="65461f6ec9f7dfb2c369c04f7b23b75bfe2b237ad10888ec005861ac845f6f80" exitCode=0
Oct 29 22:13:55.912064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:55.911702    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4" event={"ID":"20783eec-14f3-4dd3-ada8-cd52fcbe6f98","Type":"ContainerDied","Data":"65461f6ec9f7dfb2c369c04f7b23b75bfe2b237ad10888ec005861ac845f6f80"}
Oct 29 22:13:56.765655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:56.760003    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:13:57.332684 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:57.332655    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"
Oct 29 22:13:57.434751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:57.434713    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-kube-api-access-zd7pq\" (UniqueName: \"kubernetes.io/projected/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-kube-api-access-zd7pq\") pod \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\" (UID: \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\") "
Oct 29 22:13:57.434944 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:57.434804    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-openshift-ca\") pod \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\" (UID: \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\") "
Oct 29 22:13:57.434944 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:57.434836    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-xks-ca\") pod \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\" (UID: \"20783eec-14f3-4dd3-ada8-cd52fcbe6f98\") "
Oct 29 22:13:57.435373 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:57.435341    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-openshift-ca" pod "20783eec-14f3-4dd3-ada8-cd52fcbe6f98" (UID: "20783eec-14f3-4dd3-ada8-cd52fcbe6f98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:57.435703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:57.435679    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:57.438183 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:57.438158    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-kube-api-access-zd7pq" pod "20783eec-14f3-4dd3-ada8-cd52fcbe6f98" (UID: "20783eec-14f3-4dd3-ada8-cd52fcbe6f98"). InnerVolumeSpecName "kube-api-access-zd7pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:57.438320 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:57.438289    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-xks-ca" pod "20783eec-14f3-4dd3-ada8-cd52fcbe6f98" (UID: "20783eec-14f3-4dd3-ada8-cd52fcbe6f98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:57.537519 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:57.537485    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd7pq\" (UniqueName: \"kubernetes.io/projected/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-kube-api-access-zd7pq\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:57.537519 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:57.537516    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20783eec-14f3-4dd3-ada8-cd52fcbe6f98-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:13:57.968379 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:57.968346    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"
Oct 29 22:13:57.968591 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:57.968370    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4" event={"ID":"20783eec-14f3-4dd3-ada8-cd52fcbe6f98","Type":"ContainerDied","Data":"bf9d89490637821f0662f9d44d25711a373853b3c47994e2a69424b6584a6049"}
Oct 29 22:13:57.968591 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:13:57.968425    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf9d89490637821f0662f9d44d25711a373853b3c47994e2a69424b6584a6049"
Oct 29 22:13:58.671414 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:13:58.670183    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:14:00.540652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:00.539724    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:00.772788 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:00.772736    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:14:01.069379 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:01.068748    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:01.131896 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:01.131851    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:01.554710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:01.551820    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:01.554710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:01.552684    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:14:01.806803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:01.804464    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:14:01.806803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:01.805146    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:14:02.091854 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:02.091749    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:14:02.162654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:02.139801    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:02.162654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:02.144119    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:14:02.162654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:02.144196    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:14:02.610653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:02.562731    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:02.610653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:02.595566    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:14:03.026900 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:03.026801    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx" podUID="0369fe15-bc84-442e-973c-fde55e95785a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:14:03.055891 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:03.055847    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:14:03.156663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:03.132467    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.287203 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:03.159192    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.561858 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:03.560408    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.562947 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:03.562911    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.601162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:03.601120    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:14:03.622654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:03.602079    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:14:03.895654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:03.713719    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:03.895654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:03.713821    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:04.202363 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:04.150770    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:14:04.239815 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:04.239765    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2" podUID="efe88fdd-8cf4-4758-8d81-546d50c2786f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:14:04.287143 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:04.287089    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:04.400910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:04.400857    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:04.628656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:04.626867    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:14:04.632660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:04.631704    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:14:04.709692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:04.704190    7082 scope.go:122] "RemoveContainer" containerID="2c8b8a3848e94719bc55cd40292d05e14bb42368623e58bf4cce161e2afbc002"
Oct 29 22:14:04.823876 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:04.823832    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.049959 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:05.018801    7082 scope.go:122] "RemoveContainer" containerID="409b522d598d1392f782b1e8d7576d9f96260387cdca9991c67083b6e5389ea5"
Oct 29 22:14:05.049959 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:05.047071    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:05.049959 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:05.047129    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.125258 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:05.124789    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" containerID="cri-o://2b5113fa541a6181da648712b000cca709c69325c3a288335ec238cd9014119d" gracePeriod=45
Oct 29 22:14:05.230071 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:05.228954    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded"
Oct 29 22:14:05.301730 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:05.301495    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.347847 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:05.344866    7082 scope.go:122] "RemoveContainer" containerID="2f5fba3b6562539cf123efac2b9e582d85c723e4a8354e3782217909aa0c16f0"
Oct 29 22:14:05.396655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:05.393854    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": read tcp 10.134.0.2:52884->10.134.0.75:8000: read: connection reset by peer"
Oct 29 22:14:05.446172 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:05.444723    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.635298 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:05.634739    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:14:05.635866 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:05.635723    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.727670 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:05.703737    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 29 22:14:06.194028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:06.193805    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" containerID="cri-o://5bce4c30aacd93665628a05069fd9615c4e7a8a9239ae627620a80cd35a2b004" gracePeriod=44
Oct 29 22:14:06.303767 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:06.303716    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 29 22:14:06.355864 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:06.355836    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"]
Oct 29 22:14:06.457683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:06.445112    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:06.629669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:06.629082    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"]
Oct 29 22:14:06.635758 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:06.635721    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:14:06.636388 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:06.636357    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:06.707649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:06.706331    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 29 22:14:06.998658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:06.992059    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:06.998658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:06.992138    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:07.411024 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:07.390845    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:07.489427 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:07.488731    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:07.576547 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:07.576507    7082 generic.go:356] "Generic (PLEG): container finished" podID="efe88fdd-8cf4-4758-8d81-546d50c2786f" containerID="34ff6b30ffbac0f10e1d8a832bc405c73f1a8c4ac90c59517c183b0cbe080f95" exitCode=0
Oct 29 22:14:07.577199 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:07.577168    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2" event={"ID":"efe88fdd-8cf4-4758-8d81-546d50c2786f","Type":"ContainerDied","Data":"34ff6b30ffbac0f10e1d8a832bc405c73f1a8c4ac90c59517c183b0cbe080f95"}
Oct 29 22:14:07.630330 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:07.630292    7082 generic.go:356] "Generic (PLEG): container finished" podID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerID="2b5113fa541a6181da648712b000cca709c69325c3a288335ec238cd9014119d" exitCode=0
Oct 29 22:14:07.673386 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:07.672729    7082 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:07.673386 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:07.672795    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:07.673386 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:07.673198    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:07.673386 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:07.673273    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:07.942706 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:07.942585    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": dial tcp 10.134.0.75:8000: connect: connection refused"
Oct 29 22:14:07.967664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:07.955785    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded"
Oct 29 22:14:08.035694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:08.015555    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" event={"ID":"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8","Type":"ContainerDied","Data":"2b5113fa541a6181da648712b000cca709c69325c3a288335ec238cd9014119d"}
Oct 29 22:14:08.035694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:08.032047    7082 generic.go:356] "Generic (PLEG): container finished" podID="0369fe15-bc84-442e-973c-fde55e95785a" containerID="6739c9b6994f6d5bc303ce2ed054bc240cb72d5c53445eb5677cd69e878a61f5" exitCode=0
Oct 29 22:14:08.035694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:08.032140    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx" event={"ID":"0369fe15-bc84-442e-973c-fde55e95785a","Type":"ContainerDied","Data":"6739c9b6994f6d5bc303ce2ed054bc240cb72d5c53445eb5677cd69e878a61f5"}
Oct 29 22:14:08.417137 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:08.417088    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:08.515239 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:08.515191    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:08.821671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:08.821343    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:08.822182 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:08.821944    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:09.254835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.254800    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:14:09.362215 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.361876    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-home\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-home\") pod \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") "
Oct 29 22:14:09.362678 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.362657    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-model-cache\") pod \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") "
Oct 29 22:14:09.363397 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.363377    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tls-certs\") pod \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") "
Oct 29 22:14:09.363724 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.363697    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-dshm\") pod \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") "
Oct 29 22:14:09.364750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.364730    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tmp-dir\") pod \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") "
Oct 29 22:14:09.394234 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.394203    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-kube-api-access-qlw5l\" (UniqueName: \"kubernetes.io/projected/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-kube-api-access-qlw5l\") pod \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\" (UID: \"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8\") "
Oct 29 22:14:09.447009 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.441712    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-model-cache" pod "61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" (UID: "61b0b4e9-d95f-474b-a3f3-3b29bcb723d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:09.447009 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.441822    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:09.449218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.449186    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tmp-dir" pod "61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" (UID: "61b0b4e9-d95f-474b-a3f3-3b29bcb723d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:09.456770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.456738    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-home" pod "61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" (UID: "61b0b4e9-d95f-474b-a3f3-3b29bcb723d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:09.469764 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.469731    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-kube-api-access-qlw5l" pod "61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" (UID: "61b0b4e9-d95f-474b-a3f3-3b29bcb723d8"). InnerVolumeSpecName "kube-api-access-qlw5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:09.508777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.508742    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlw5l\" (UniqueName: \"kubernetes.io/projected/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-kube-api-access-qlw5l\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:09.509014 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.508998    7082 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:09.509108 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.509097    7082 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:09.509190 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.509181    7082 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:09.530058 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.530014    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:09.806268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.806227    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:09.845861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.845336    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:14:09.867655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.849901    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:09.867655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.851729    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-dshm" pod "61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" (UID: "61b0b4e9-d95f-474b-a3f3-3b29bcb723d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:09.867655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.851798    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tls-certs" pod "61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" (UID: "61b0b4e9-d95f-474b-a3f3-3b29bcb723d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:09.880649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.878426    7082 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:09.880649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.878488    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:09.936671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.919189    7082 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:09.951720 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:09.951687    7082 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:10.373536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:10.373505    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:14:10.465033 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:10.442618    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:10.523452 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:10.487790    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:14:10.925653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:10.918283    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:10.948909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:10.948874    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"
Oct 29 22:14:10.999664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:10.990278    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz" event={"ID":"61b0b4e9-d95f-474b-a3f3-3b29bcb723d8","Type":"ContainerDied","Data":"0b97749f66c751b7acf54e229e30a535d2c20c69a3f3cbe57fb8c950db8c7cc2"}
Oct 29 22:14:10.999664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:10.990354    7082 scope.go:122] "RemoveContainer" containerID="2b5113fa541a6181da648712b000cca709c69325c3a288335ec238cd9014119d"
Oct 29 22:14:11.381660 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.364794    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr" podUID="e3a92df7-921d-4f5f-a18b-0c9494dfba37" containerName="payload-processing" containerID="cri-o://2182d653f74f40e443dd8498d2133d0df2fbf2720f27132ca808221758b13d17" gracePeriod=25
Oct 29 22:14:11.385730 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.385696    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"]
Oct 29 22:14:11.393655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.386168    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5" podUID="e175ca6e-a9dd-4596-9dbf-bba706737b2e" containerName="payload-processing" containerID="cri-o://18f51fbf625273184a657d420d27441e1580b71e0a23fc19236ee50c9c14e33f" gracePeriod=30
Oct 29 22:14:11.396283 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.379861    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"
Oct 29 22:14:11.396283 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.392811    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:11.396283 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.396174    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:11.506227 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.480663    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:11.506227 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.505740    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:11.583711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.581284    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": dial tcp 10.134.0.88:8000: connect: connection refused"
Oct 29 22:14:11.585895 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.585853    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q"
Oct 29 22:14:11.610953 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.603570    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"
Oct 29 22:14:11.629411 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.629360    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk" podUID="b742a630-e75b-4040-8521-12388f3d0427" containerName="payload-pre-processing" containerID="cri-o://18cc190494e31d0dfacc0b5498f1f7da571342517d08184baeae71bd2d5883d8" gracePeriod=25
Oct 29 22:14:11.708401 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.662609    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:11.739590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.685697    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"]
Oct 29 22:14:11.739590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.721255    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp" podUID="ba49c3e6-27f2-4b68-b0a4-5c83c089395c" containerName="payload-pre-processing" containerID="cri-o://392c768407dd8001f730fe1d55694a0ba1114b8f99c377a5dc7a9ac2f682f32a" gracePeriod=30
Oct 29 22:14:11.739590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.686167    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe88fdd-8cf4-4758-8d81-546d50c2786f-kube-api-access-kq7wz\" (UniqueName: \"kubernetes.io/projected/efe88fdd-8cf4-4758-8d81-546d50c2786f-kube-api-access-kq7wz\") pod \"efe88fdd-8cf4-4758-8d81-546d50c2786f\" (UID: \"efe88fdd-8cf4-4758-8d81-546d50c2786f\") "
Oct 29 22:14:11.739590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.721946    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0369fe15-bc84-442e-973c-fde55e95785a-kube-api-access-qmd27\" (UniqueName: \"kubernetes.io/projected/0369fe15-bc84-442e-973c-fde55e95785a-kube-api-access-qmd27\") pod \"0369fe15-bc84-442e-973c-fde55e95785a\" (UID: \"0369fe15-bc84-442e-973c-fde55e95785a\") "
Oct 29 22:14:11.739590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.721987    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efe88fdd-8cf4-4758-8d81-546d50c2786f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe88fdd-8cf4-4758-8d81-546d50c2786f-plugins-config-volume\") pod \"efe88fdd-8cf4-4758-8d81-546d50c2786f\" (UID: \"efe88fdd-8cf4-4758-8d81-546d50c2786f\") "
Oct 29 22:14:11.739590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.733845    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe88fdd-8cf4-4758-8d81-546d50c2786f-plugins-config-volume" pod "efe88fdd-8cf4-4758-8d81-546d50c2786f" (UID: "efe88fdd-8cf4-4758-8d81-546d50c2786f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:11.785006 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.784973    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0369fe15-bc84-442e-973c-fde55e95785a-kube-api-access-qmd27" pod "0369fe15-bc84-442e-973c-fde55e95785a" (UID: "0369fe15-bc84-442e-973c-fde55e95785a"). InnerVolumeSpecName "kube-api-access-qmd27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:11.785881 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.785846    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe88fdd-8cf4-4758-8d81-546d50c2786f-kube-api-access-kq7wz" pod "efe88fdd-8cf4-4758-8d81-546d50c2786f" (UID: "efe88fdd-8cf4-4758-8d81-546d50c2786f"). InnerVolumeSpecName "kube-api-access-kq7wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:11.833177 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.828308    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0369fe15-bc84-442e-973c-fde55e95785a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0369fe15-bc84-442e-973c-fde55e95785a-plugins-config-volume\") pod \"0369fe15-bc84-442e-973c-fde55e95785a\" (UID: \"0369fe15-bc84-442e-973c-fde55e95785a\") "
Oct 29 22:14:11.839028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.838826    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmd27\" (UniqueName: \"kubernetes.io/projected/0369fe15-bc84-442e-973c-fde55e95785a-kube-api-access-qmd27\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:11.839028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.838862    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efe88fdd-8cf4-4758-8d81-546d50c2786f-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:11.839028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.838878    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq7wz\" (UniqueName: \"kubernetes.io/projected/efe88fdd-8cf4-4758-8d81-546d50c2786f-kube-api-access-kq7wz\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:11.840823 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.839607    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0369fe15-bc84-442e-973c-fde55e95785a-plugins-config-volume" pod "0369fe15-bc84-442e-973c-fde55e95785a" (UID: "0369fe15-bc84-442e-973c-fde55e95785a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:11.840823 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.839738    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"]
Oct 29 22:14:11.871407 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.871370    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b80d5b-qnxwx"]
Oct 29 22:14:11.899538 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.899408    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"]
Oct 29 22:14:11.911550 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.911518    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/simulator-w6-a940bc-kserve-68c7c9f47d-qkbzz"]
Oct 29 22:14:11.932366 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.932330    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx" event={"ID":"0369fe15-bc84-442e-973c-fde55e95785a","Type":"ContainerDied","Data":"16f13b5e73624452f515e77cc4729901f3c92f5ad8ccd55081e00e497e2a326f"}
Oct 29 22:14:11.932943 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.932920    7082 scope.go:122] "RemoveContainer" containerID="6739c9b6994f6d5bc303ce2ed054bc240cb72d5c53445eb5677cd69e878a61f5"
Oct 29 22:14:11.933169 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.933152    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"
Oct 29 22:14:11.940445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.940407    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0369fe15-bc84-442e-973c-fde55e95785a-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:11.952036 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.949539    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2" event={"ID":"efe88fdd-8cf4-4758-8d81-546d50c2786f","Type":"ContainerDied","Data":"418e3a32ff266ea48e4dc95898e6651460716017a6211f67d929fb1c75fab2ce"}
Oct 29 22:14:11.952036 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.949693    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"
Oct 29 22:14:11.957782 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.956032    7082 generic.go:356] "Generic (PLEG): container finished" podID="e175ca6e-a9dd-4596-9dbf-bba706737b2e" containerID="18f51fbf625273184a657d420d27441e1580b71e0a23fc19236ee50c9c14e33f" exitCode=0
Oct 29 22:14:11.957782 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.956135    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5" event={"ID":"e175ca6e-a9dd-4596-9dbf-bba706737b2e","Type":"ContainerDied","Data":"18f51fbf625273184a657d420d27441e1580b71e0a23fc19236ee50c9c14e33f"}
Oct 29 22:14:11.962589 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.962530    7082 generic.go:356] "Generic (PLEG): container finished" podID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerID="5bce4c30aacd93665628a05069fd9615c4e7a8a9239ae627620a80cd35a2b004" exitCode=0
Oct 29 22:14:11.963609 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:11.962750    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" event={"ID":"09277a7d-b7f4-4584-b6d6-8919612bcc77","Type":"ContainerDied","Data":"5bce4c30aacd93665628a05069fd9615c4e7a8a9239ae627620a80cd35a2b004"}
Oct 29 22:14:12.049991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.049947    7082 scope.go:122] "RemoveContainer" containerID="34ff6b30ffbac0f10e1d8a832bc405c73f1a8c4ac90c59517c183b0cbe080f95"
Oct 29 22:14:12.100793 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.100741    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"]
Oct 29 22:14:12.108098 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.107962    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b31f3f-6fb8674d76-whjf2"]
Oct 29 22:14:12.146370 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.145691    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"]
Oct 29 22:14:12.158694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.154867    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b31f3f-5dff599f7f-29wxx"]
Oct 29 22:14:12.247804 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.247656    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"
Oct 29 22:14:12.250425 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.250370    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:14:12.253742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.253713    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"
Oct 29 22:14:12.365650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.364243    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-home\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-home\") pod \"09277a7d-b7f4-4584-b6d6-8919612bcc77\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") "
Oct 29 22:14:12.365650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.364344    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e175ca6e-a9dd-4596-9dbf-bba706737b2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e175ca6e-a9dd-4596-9dbf-bba706737b2e-plugins-config-volume\") pod \"e175ca6e-a9dd-4596-9dbf-bba706737b2e\" (UID: \"e175ca6e-a9dd-4596-9dbf-bba706737b2e\") "
Oct 29 22:14:12.365650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.364394    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e175ca6e-a9dd-4596-9dbf-bba706737b2e-kube-api-access-dmnhx\" (UniqueName: \"kubernetes.io/projected/e175ca6e-a9dd-4596-9dbf-bba706737b2e-kube-api-access-dmnhx\") pod \"e175ca6e-a9dd-4596-9dbf-bba706737b2e\" (UID: \"e175ca6e-a9dd-4596-9dbf-bba706737b2e\") "
Oct 29 22:14:12.365650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.364425    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09277a7d-b7f4-4584-b6d6-8919612bcc77-kube-api-access-8qv85\" (UniqueName: \"kubernetes.io/projected/09277a7d-b7f4-4584-b6d6-8919612bcc77-kube-api-access-8qv85\") pod \"09277a7d-b7f4-4584-b6d6-8919612bcc77\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") "
Oct 29 22:14:12.365650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.364504    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-dshm\") pod \"09277a7d-b7f4-4584-b6d6-8919612bcc77\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") "
Oct 29 22:14:12.365650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.364584    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-model-cache\") pod \"09277a7d-b7f4-4584-b6d6-8919612bcc77\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") "
Oct 29 22:14:12.365650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.364701    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09277a7d-b7f4-4584-b6d6-8919612bcc77-tls-certs\" (UniqueName: \"kubernetes.io/secret/09277a7d-b7f4-4584-b6d6-8919612bcc77-tls-certs\") pod \"09277a7d-b7f4-4584-b6d6-8919612bcc77\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") "
Oct 29 22:14:12.365650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.364818    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-kube-api-access-bt4n4\" (UniqueName: \"kubernetes.io/projected/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-kube-api-access-bt4n4\") pod \"ba49c3e6-27f2-4b68-b0a4-5c83c089395c\" (UID: \"ba49c3e6-27f2-4b68-b0a4-5c83c089395c\") "
Oct 29 22:14:12.365650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.364911    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-plugins-config-volume\") pod \"ba49c3e6-27f2-4b68-b0a4-5c83c089395c\" (UID: \"ba49c3e6-27f2-4b68-b0a4-5c83c089395c\") "
Oct 29 22:14:12.365650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.364969    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-tmp-dir\") pod \"09277a7d-b7f4-4584-b6d6-8919612bcc77\" (UID: \"09277a7d-b7f4-4584-b6d6-8919612bcc77\") "
Oct 29 22:14:12.373286 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.370945    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-home" pod "09277a7d-b7f4-4584-b6d6-8919612bcc77" (UID: "09277a7d-b7f4-4584-b6d6-8919612bcc77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:12.375153 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.373541    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-model-cache" pod "09277a7d-b7f4-4584-b6d6-8919612bcc77" (UID: "09277a7d-b7f4-4584-b6d6-8919612bcc77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:12.375153 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.373759    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e175ca6e-a9dd-4596-9dbf-bba706737b2e-plugins-config-volume" pod "e175ca6e-a9dd-4596-9dbf-bba706737b2e" (UID: "e175ca6e-a9dd-4596-9dbf-bba706737b2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:12.375153 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.374571    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-plugins-config-volume" pod "ba49c3e6-27f2-4b68-b0a4-5c83c089395c" (UID: "ba49c3e6-27f2-4b68-b0a4-5c83c089395c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:12.375153 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.374947    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-tmp-dir" pod "09277a7d-b7f4-4584-b6d6-8919612bcc77" (UID: "09277a7d-b7f4-4584-b6d6-8919612bcc77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:12.377993 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.377970    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09277a7d-b7f4-4584-b6d6-8919612bcc77-kube-api-access-8qv85" pod "09277a7d-b7f4-4584-b6d6-8919612bcc77" (UID: "09277a7d-b7f4-4584-b6d6-8919612bcc77"). InnerVolumeSpecName "kube-api-access-8qv85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:12.378269 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.378234    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e175ca6e-a9dd-4596-9dbf-bba706737b2e-kube-api-access-dmnhx" pod "e175ca6e-a9dd-4596-9dbf-bba706737b2e" (UID: "e175ca6e-a9dd-4596-9dbf-bba706737b2e"). InnerVolumeSpecName "kube-api-access-dmnhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:12.380188 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.380164    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-dshm" pod "09277a7d-b7f4-4584-b6d6-8919612bcc77" (UID: "09277a7d-b7f4-4584-b6d6-8919612bcc77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:12.380517 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.380492    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09277a7d-b7f4-4584-b6d6-8919612bcc77-tls-certs" pod "09277a7d-b7f4-4584-b6d6-8919612bcc77" (UID: "09277a7d-b7f4-4584-b6d6-8919612bcc77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:12.380692 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.380507    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-kube-api-access-bt4n4" pod "ba49c3e6-27f2-4b68-b0a4-5c83c089395c" (UID: "ba49c3e6-27f2-4b68-b0a4-5c83c089395c"). InnerVolumeSpecName "kube-api-access-bt4n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:12.467277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.467198    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e175ca6e-a9dd-4596-9dbf-bba706737b2e-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:12.467277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.467240    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmnhx\" (UniqueName: \"kubernetes.io/projected/e175ca6e-a9dd-4596-9dbf-bba706737b2e-kube-api-access-dmnhx\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:12.467277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.467255    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qv85\" (UniqueName: \"kubernetes.io/projected/09277a7d-b7f4-4584-b6d6-8919612bcc77-kube-api-access-8qv85\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:12.467277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.467270    7082 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:12.467512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.467284    7082 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:12.467512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.467297    7082 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09277a7d-b7f4-4584-b6d6-8919612bcc77-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:12.467512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.467309    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt4n4\" (UniqueName: \"kubernetes.io/projected/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-kube-api-access-bt4n4\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:12.467512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.467321    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba49c3e6-27f2-4b68-b0a4-5c83c089395c-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:12.467512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.467360    7082 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:12.467512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:12.467373    7082 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09277a7d-b7f4-4584-b6d6-8919612bcc77-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:13.037363 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.029356    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:13.138898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.137961    7082 generic.go:356] "Generic (PLEG): container finished" podID="ba49c3e6-27f2-4b68-b0a4-5c83c089395c" containerID="392c768407dd8001f730fe1d55694a0ba1114b8f99c377a5dc7a9ac2f682f32a" exitCode=0
Oct 29 22:14:13.138898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.138198    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp" event={"ID":"ba49c3e6-27f2-4b68-b0a4-5c83c089395c","Type":"ContainerDied","Data":"392c768407dd8001f730fe1d55694a0ba1114b8f99c377a5dc7a9ac2f682f32a"}
Oct 29 22:14:13.138898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.138302    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp" event={"ID":"ba49c3e6-27f2-4b68-b0a4-5c83c089395c","Type":"ContainerDied","Data":"30a64f24738d1e0ad43127e331339581416aaf3727d79669640c447a84c554ae"}
Oct 29 22:14:13.138898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.138334    7082 scope.go:122] "RemoveContainer" containerID="392c768407dd8001f730fe1d55694a0ba1114b8f99c377a5dc7a9ac2f682f32a"
Oct 29 22:14:13.138898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.138469    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"
Oct 29 22:14:13.145456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.145424    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5" event={"ID":"e175ca6e-a9dd-4596-9dbf-bba706737b2e","Type":"ContainerDied","Data":"c01e3ad703481f9da97e9fb3b3f835a96e7879be2bcd68a6584e4cb46fce9418"}
Oct 29 22:14:13.145884 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.145813    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"
Oct 29 22:14:13.155993 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.155964    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl" event={"ID":"09277a7d-b7f4-4584-b6d6-8919612bcc77","Type":"ContainerDied","Data":"11ad5fb97818fd03fe8f20f9e3d1a2d749a422c3a5fed62728ccfd1117ef8063"}
Oct 29 22:14:13.156113 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.156038    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"
Oct 29 22:14:13.162095 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.162079    7082 scope.go:122] "RemoveContainer" containerID="392c768407dd8001f730fe1d55694a0ba1114b8f99c377a5dc7a9ac2f682f32a"
Oct 29 22:14:13.162600 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:13.162571    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392c768407dd8001f730fe1d55694a0ba1114b8f99c377a5dc7a9ac2f682f32a\": container with ID starting with 392c768407dd8001f730fe1d55694a0ba1114b8f99c377a5dc7a9ac2f682f32a not found: ID does not exist" containerID="392c768407dd8001f730fe1d55694a0ba1114b8f99c377a5dc7a9ac2f682f32a"
Oct 29 22:14:13.162778 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.162754    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392c768407dd8001f730fe1d55694a0ba1114b8f99c377a5dc7a9ac2f682f32a"} err="failed to get container status \"392c768407dd8001f730fe1d55694a0ba1114b8f99c377a5dc7a9ac2f682f32a\": rpc error: code = NotFound desc = could not find container \"392c768407dd8001f730fe1d55694a0ba1114b8f99c377a5dc7a9ac2f682f32a\": container with ID starting with 392c768407dd8001f730fe1d55694a0ba1114b8f99c377a5dc7a9ac2f682f32a not found: ID does not exist"
Oct 29 22:14:13.162890 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.162879    7082 scope.go:122] "RemoveContainer" containerID="18f51fbf625273184a657d420d27441e1580b71e0a23fc19236ee50c9c14e33f"
Oct 29 22:14:13.176988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.176971    7082 scope.go:122] "RemoveContainer" containerID="5bce4c30aacd93665628a05069fd9615c4e7a8a9239ae627620a80cd35a2b004"
Oct 29 22:14:13.229103 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.229074    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"]
Oct 29 22:14:13.239875 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.239849    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a940bc/distinct-w6-a940bc-kserve-55cc665cb6-mm9xl"]
Oct 29 22:14:13.269032 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.269004    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"]
Oct 29 22:14:13.274523 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.274503    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-547d67599f-xfgsp"]
Oct 29 22:14:13.308726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.308704    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"]
Oct 29 22:14:13.314131 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.314107    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b80d5b-69f7db55b4-5mbc5"]
Oct 29 22:14:13.693045 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.692951    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0369fe15-bc84-442e-973c-fde55e95785a" path="/var/lib/kubelet/pods/0369fe15-bc84-442e-973c-fde55e95785a/volumes"
Oct 29 22:14:13.694184 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.694139    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" path="/var/lib/kubelet/pods/09277a7d-b7f4-4584-b6d6-8919612bcc77/volumes"
Oct 29 22:14:13.695305 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.695283    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" path="/var/lib/kubelet/pods/61b0b4e9-d95f-474b-a3f3-3b29bcb723d8/volumes"
Oct 29 22:14:13.696804 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.696776    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a169dae-6fcb-4ce3-8132-b00da958bd19" path="/var/lib/kubelet/pods/9a169dae-6fcb-4ce3-8132-b00da958bd19/volumes"
Oct 29 22:14:13.697893 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.697872    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba49c3e6-27f2-4b68-b0a4-5c83c089395c" path="/var/lib/kubelet/pods/ba49c3e6-27f2-4b68-b0a4-5c83c089395c/volumes"
Oct 29 22:14:13.699066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.699046    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e175ca6e-a9dd-4596-9dbf-bba706737b2e" path="/var/lib/kubelet/pods/e175ca6e-a9dd-4596-9dbf-bba706737b2e/volumes"
Oct 29 22:14:13.699838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:13.699819    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efe88fdd-8cf4-4758-8d81-546d50c2786f" path="/var/lib/kubelet/pods/efe88fdd-8cf4-4758-8d81-546d50c2786f/volumes"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.646580    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"]
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654430    7082 cpu_manager.go:414] "containerMap: removing container" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654466    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654514    7082 cpu_manager.go:414] "containerMap: removing container" podUID="20783eec-14f3-4dd3-ada8-cd52fcbe6f98" containerName="revoke-keys"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654522    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20783eec-14f3-4dd3-ada8-cd52fcbe6f98" containerName="revoke-keys"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654541    7082 cpu_manager.go:414] "containerMap: removing container" podUID="efe88fdd-8cf4-4758-8d81-546d50c2786f" containerName="payload-processing"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654549    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe88fdd-8cf4-4758-8d81-546d50c2786f" containerName="payload-processing"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654559    7082 cpu_manager.go:414] "containerMap: removing container" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654566    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654587    7082 cpu_manager.go:414] "containerMap: removing container" podUID="ba49c3e6-27f2-4b68-b0a4-5c83c089395c" containerName="payload-pre-processing"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654595    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba49c3e6-27f2-4b68-b0a4-5c83c089395c" containerName="payload-pre-processing"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654614    7082 cpu_manager.go:414] "containerMap: removing container" podUID="e175ca6e-a9dd-4596-9dbf-bba706737b2e" containerName="payload-processing"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654621    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e175ca6e-a9dd-4596-9dbf-bba706737b2e" containerName="payload-processing"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654662    7082 cpu_manager.go:414] "containerMap: removing container" podUID="0369fe15-bc84-442e-973c-fde55e95785a" containerName="payload-pre-processing"
Oct 29 22:14:14.654663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654670    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0369fe15-bc84-442e-973c-fde55e95785a" containerName="payload-pre-processing"
Oct 29 22:14:14.656262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654872    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe88fdd-8cf4-4758-8d81-546d50c2786f" containerName="payload-processing"
Oct 29 22:14:14.656262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654883    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b0b4e9-d95f-474b-a3f3-3b29bcb723d8" containerName="main"
Oct 29 22:14:14.656262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654893    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba49c3e6-27f2-4b68-b0a4-5c83c089395c" containerName="payload-pre-processing"
Oct 29 22:14:14.656262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654906    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="2702b7ee-1ae0-4831-92d4-ae6d7bb408c6" containerName="istio-proxy"
Oct 29 22:14:14.656262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654915    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="e175ca6e-a9dd-4596-9dbf-bba706737b2e" containerName="payload-processing"
Oct 29 22:14:14.656262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654928    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="0369fe15-bc84-442e-973c-fde55e95785a" containerName="payload-pre-processing"
Oct 29 22:14:14.656262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654941    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="09277a7d-b7f4-4584-b6d6-8919612bcc77" containerName="main"
Oct 29 22:14:14.656262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.654954    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="20783eec-14f3-4dd3-ada8-cd52fcbe6f98" containerName="revoke-keys"
Oct 29 22:14:14.735688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.733071    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"
Oct 29 22:14:14.762162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.755422    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:14:14.762162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.756364    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:14:14.762162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.756462    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:14:14.762162 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.756547    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:14:14.765720 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.765692    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"]
Oct 29 22:14:14.813057 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.803882    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"
Oct 29 22:14:14.820809 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.820781    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"]
Oct 29 22:14:14.841621 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.841591    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"]
Oct 29 22:14:14.844840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.844812    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98ad10af-94rzf\" (UID: \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"
Oct 29 22:14:14.844966 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.844908    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m5xd\" (UniqueName: \"kubernetes.io/projected/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-kube-api-access-5m5xd\") pod \"maas-api-revoke-sub-98ad10af-94rzf\" (UID: \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"
Oct 29 22:14:14.844966 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.844952    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98ad10af-94rzf\" (UID: \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"
Oct 29 22:14:14.948260 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.948173    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98ad10af-94rzf\" (UID: \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"
Oct 29 22:14:14.948420 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.948259    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m5xd\" (UniqueName: \"kubernetes.io/projected/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-kube-api-access-5m5xd\") pod \"maas-api-revoke-sub-98ad10af-94rzf\" (UID: \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"
Oct 29 22:14:14.948420 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.948293    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98ad10af-94rzf\" (UID: \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"
Oct 29 22:14:14.948420 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.948373    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eecee06-ztv4z\" (UID: \"7fa68641-c5bc-46e1-8492-f46183338fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"
Oct 29 22:14:14.948605 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.948437    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eecee06-ztv4z\" (UID: \"7fa68641-c5bc-46e1-8492-f46183338fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"
Oct 29 22:14:14.948605 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.948471    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mght\" (UniqueName: \"kubernetes.io/projected/7fa68641-c5bc-46e1-8492-f46183338fe7-kube-api-access-8mght\") pod \"maas-api-revoke-sub-0eecee06-ztv4z\" (UID: \"7fa68641-c5bc-46e1-8492-f46183338fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"
Oct 29 22:14:14.948837 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.948811    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98ad10af-94rzf\" (UID: \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"
Oct 29 22:14:14.950867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.950847    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98ad10af-94rzf\" (UID: \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"
Oct 29 22:14:14.971533 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:14.971503    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m5xd\" (UniqueName: \"kubernetes.io/projected/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-kube-api-access-5m5xd\") pod \"maas-api-revoke-sub-98ad10af-94rzf\" (UID: \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"
Oct 29 22:14:15.050306 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:15.050265    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eecee06-ztv4z\" (UID: \"7fa68641-c5bc-46e1-8492-f46183338fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"
Oct 29 22:14:15.050473 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:15.050332    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eecee06-ztv4z\" (UID: \"7fa68641-c5bc-46e1-8492-f46183338fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"
Oct 29 22:14:15.050473 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:15.050357    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mght\" (UniqueName: \"kubernetes.io/projected/7fa68641-c5bc-46e1-8492-f46183338fe7-kube-api-access-8mght\") pod \"maas-api-revoke-sub-0eecee06-ztv4z\" (UID: \"7fa68641-c5bc-46e1-8492-f46183338fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"
Oct 29 22:14:15.051050 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:15.051026    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eecee06-ztv4z\" (UID: \"7fa68641-c5bc-46e1-8492-f46183338fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"
Oct 29 22:14:15.052838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:15.052819    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eecee06-ztv4z\" (UID: \"7fa68641-c5bc-46e1-8492-f46183338fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"
Oct 29 22:14:15.072923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:15.072899    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mght\" (UniqueName: \"kubernetes.io/projected/7fa68641-c5bc-46e1-8492-f46183338fe7-kube-api-access-8mght\") pod \"maas-api-revoke-sub-0eecee06-ztv4z\" (UID: \"7fa68641-c5bc-46e1-8492-f46183338fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"
Oct 29 22:14:15.126682 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:15.126606    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"
Oct 29 22:14:15.151731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:15.151703    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"
Oct 29 22:14:15.656373 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:15.656345    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"]
Oct 29 22:14:15.656734 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:14:15.656539    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f43c8d_9dac_4f7d_94e1_e0100cd449d0.slice/crio-8eb79e6364054338e06033dca5ed12cf4c70a08265c5fef61b60e17af1cf45af WatchSource:0}: Error finding container 8eb79e6364054338e06033dca5ed12cf4c70a08265c5fef61b60e17af1cf45af: Status 404 returned error can't find the container with id 8eb79e6364054338e06033dca5ed12cf4c70a08265c5fef61b60e17af1cf45af
Oct 29 22:14:15.714087 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:14:15.714062    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa68641_c5bc_46e1_8492_f46183338fe7.slice/crio-96948f7dda3e459811bc6c4ddc5e8fdee0ef125dace1c61b097a08a3660b3ebb WatchSource:0}: Error finding container 96948f7dda3e459811bc6c4ddc5e8fdee0ef125dace1c61b097a08a3660b3ebb: Status 404 returned error can't find the container with id 96948f7dda3e459811bc6c4ddc5e8fdee0ef125dace1c61b097a08a3660b3ebb
Oct 29 22:14:15.715902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:15.715872    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"]
Oct 29 22:14:16.500395 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:16.500357    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z" event={"ID":"7fa68641-c5bc-46e1-8492-f46183338fe7","Type":"ContainerStarted","Data":"26ca767b609c39363b1c786977aab03400f8a5e27c30339b8d8d9d8f7ea10f18"}
Oct 29 22:14:16.500714 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:16.500686    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z" event={"ID":"7fa68641-c5bc-46e1-8492-f46183338fe7","Type":"ContainerStarted","Data":"96948f7dda3e459811bc6c4ddc5e8fdee0ef125dace1c61b097a08a3660b3ebb"}
Oct 29 22:14:16.502472 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:16.502440    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf" event={"ID":"59f43c8d-9dac-4f7d-94e1-e0100cd449d0","Type":"ContainerStarted","Data":"7d18156dbe981e7a09a9949d0fa73d9f442438f43cf85934be74fb9875bda2c8"}
Oct 29 22:14:16.502584 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:16.502488    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf" event={"ID":"59f43c8d-9dac-4f7d-94e1-e0100cd449d0","Type":"ContainerStarted","Data":"8eb79e6364054338e06033dca5ed12cf4c70a08265c5fef61b60e17af1cf45af"}
Oct 29 22:14:16.544170 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:16.544108    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z" podStartSLOduration=2.54408999 podStartE2EDuration="2.54408999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:16.540027667 +0000 UTC m=+1629.040069970" watchObservedRunningTime="2026-10-29 22:14:16.54408999 +0000 UTC m=+1629.044132285"
Oct 29 22:14:17.579252 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:17.578305    7082 generic.go:356] "Generic (PLEG): container finished" podID="59f43c8d-9dac-4f7d-94e1-e0100cd449d0" containerID="7d18156dbe981e7a09a9949d0fa73d9f442438f43cf85934be74fb9875bda2c8" exitCode=0
Oct 29 22:14:17.579252 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:17.579065    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf" event={"ID":"59f43c8d-9dac-4f7d-94e1-e0100cd449d0","Type":"ContainerDied","Data":"7d18156dbe981e7a09a9949d0fa73d9f442438f43cf85934be74fb9875bda2c8"}
Oct 29 22:14:17.609455 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:17.605220    7082 generic.go:356] "Generic (PLEG): container finished" podID="7fa68641-c5bc-46e1-8492-f46183338fe7" containerID="26ca767b609c39363b1c786977aab03400f8a5e27c30339b8d8d9d8f7ea10f18" exitCode=0
Oct 29 22:14:17.609455 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:17.605327    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z" event={"ID":"7fa68641-c5bc-46e1-8492-f46183338fe7","Type":"ContainerDied","Data":"26ca767b609c39363b1c786977aab03400f8a5e27c30339b8d8d9d8f7ea10f18"}
Oct 29 22:14:17.892332 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:17.892247    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"]
Oct 29 22:14:17.903082 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:17.903039    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b31f3f-dgnlg"]
Oct 29 22:14:18.264771 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:18.264696    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv_def131f2-c87a-4d18-9ad3-144bfc1d7f67/payload-processing/0.log"
Oct 29 22:14:18.488233 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:18.488201    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv_def131f2-c87a-4d18-9ad3-144bfc1d7f67/payload-processing/0.log"
Oct 29 22:14:18.881288 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:18.881264    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"
Oct 29 22:14:18.884413 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:18.884378    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk" podUID="b742a630-e75b-4040-8521-12388f3d0427" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:14:18.885245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:18.885221    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"
Oct 29 22:14:18.996701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:18.996668    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"]
Oct 29 22:14:18.997598 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:18.997566    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy" containerID="cri-o://d0d60ef12968e8110a371e797d6943db767c0e8451bea3dfbfa06998a851c6ee" gracePeriod=30
Oct 29 22:14:19.029125 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.029100    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-xks-ca\") pod \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\" (UID: \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\") "
Oct 29 22:14:19.029245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.029145    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-openshift-ca\") pod \"7fa68641-c5bc-46e1-8492-f46183338fe7\" (UID: \"7fa68641-c5bc-46e1-8492-f46183338fe7\") "
Oct 29 22:14:19.029245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.029186    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa68641-c5bc-46e1-8492-f46183338fe7-kube-api-access-8mght\" (UniqueName: \"kubernetes.io/projected/7fa68641-c5bc-46e1-8492-f46183338fe7-kube-api-access-8mght\") pod \"7fa68641-c5bc-46e1-8492-f46183338fe7\" (UID: \"7fa68641-c5bc-46e1-8492-f46183338fe7\") "
Oct 29 22:14:19.029374 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.029244    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-xks-ca\") pod \"7fa68641-c5bc-46e1-8492-f46183338fe7\" (UID: \"7fa68641-c5bc-46e1-8492-f46183338fe7\") "
Oct 29 22:14:19.029374 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.029332    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-kube-api-access-5m5xd\" (UniqueName: \"kubernetes.io/projected/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-kube-api-access-5m5xd\") pod \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\" (UID: \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\") "
Oct 29 22:14:19.029497 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.029423    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-openshift-ca\") pod \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\" (UID: \"59f43c8d-9dac-4f7d-94e1-e0100cd449d0\") "
Oct 29 22:14:19.029611 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.029590    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-openshift-ca" pod "7fa68641-c5bc-46e1-8492-f46183338fe7" (UID: "7fa68641-c5bc-46e1-8492-f46183338fe7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:19.030005 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.029982    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-openshift-ca" pod "59f43c8d-9dac-4f7d-94e1-e0100cd449d0" (UID: "59f43c8d-9dac-4f7d-94e1-e0100cd449d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:19.030272 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.030191    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:19.030272 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.030204    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:19.032389 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.032346    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-kube-api-access-5m5xd" pod "59f43c8d-9dac-4f7d-94e1-e0100cd449d0" (UID: "59f43c8d-9dac-4f7d-94e1-e0100cd449d0"). InnerVolumeSpecName "kube-api-access-5m5xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:19.039695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.039672    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-xks-ca" pod "59f43c8d-9dac-4f7d-94e1-e0100cd449d0" (UID: "59f43c8d-9dac-4f7d-94e1-e0100cd449d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:19.039808 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.039741    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa68641-c5bc-46e1-8492-f46183338fe7-kube-api-access-8mght" pod "7fa68641-c5bc-46e1-8492-f46183338fe7" (UID: "7fa68641-c5bc-46e1-8492-f46183338fe7"). InnerVolumeSpecName "kube-api-access-8mght". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:19.041899 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.041874    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-xks-ca" pod "7fa68641-c5bc-46e1-8492-f46183338fe7" (UID: "7fa68641-c5bc-46e1-8492-f46183338fe7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:19.150748 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.144235    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa68641-c5bc-46e1-8492-f46183338fe7-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:19.150748 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.144267    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m5xd\" (UniqueName: \"kubernetes.io/projected/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-kube-api-access-5m5xd\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:19.150748 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.144282    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f43c8d-9dac-4f7d-94e1-e0100cd449d0-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:19.150748 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.144296    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mght\" (UniqueName: \"kubernetes.io/projected/7fa68641-c5bc-46e1-8492-f46183338fe7-kube-api-access-8mght\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:19.251579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.217691    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"]
Oct 29 22:14:19.251579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.218845    7082 cpu_manager.go:414] "containerMap: removing container" podUID="7fa68641-c5bc-46e1-8492-f46183338fe7" containerName="revoke-keys"
Oct 29 22:14:19.251579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.218870    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa68641-c5bc-46e1-8492-f46183338fe7" containerName="revoke-keys"
Oct 29 22:14:19.251579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.218886    7082 cpu_manager.go:414] "containerMap: removing container" podUID="59f43c8d-9dac-4f7d-94e1-e0100cd449d0" containerName="revoke-keys"
Oct 29 22:14:19.251579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.218894    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f43c8d-9dac-4f7d-94e1-e0100cd449d0" containerName="revoke-keys"
Oct 29 22:14:19.251579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.219080    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa68641-c5bc-46e1-8492-f46183338fe7" containerName="revoke-keys"
Oct 29 22:14:19.251579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.219096    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f43c8d-9dac-4f7d-94e1-e0100cd449d0" containerName="revoke-keys"
Oct 29 22:14:19.274657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.258084    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"
Oct 29 22:14:19.363655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.363116    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8\" (UID: \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"
Oct 29 22:14:19.363655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.363165    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9knk\" (UniqueName: \"kubernetes.io/projected/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-kube-api-access-j9knk\") pod \"maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8\" (UID: \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"
Oct 29 22:14:19.382102 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.376368    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8\" (UID: \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"
Oct 29 22:14:19.406769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.406602    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"]
Oct 29 22:14:19.478511 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.478476    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8\" (UID: \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"
Oct 29 22:14:19.478773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.478750    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8\" (UID: \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"
Oct 29 22:14:19.478853 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.478794    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9knk\" (UniqueName: \"kubernetes.io/projected/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-kube-api-access-j9knk\") pod \"maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8\" (UID: \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"
Oct 29 22:14:19.479465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.479445    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8\" (UID: \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"
Oct 29 22:14:19.480967 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.480941    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8\" (UID: \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"
Oct 29 22:14:19.506870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.506837    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9knk\" (UniqueName: \"kubernetes.io/projected/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-kube-api-access-j9knk\") pod \"maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8\" (UID: \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"
Oct 29 22:14:19.611289 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.611250    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"]
Oct 29 22:14:19.616033 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.616012    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"
Oct 29 22:14:19.629402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.629379    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z"
Oct 29 22:14:19.631340 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.631321    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf"
Oct 29 22:14:19.634716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.634692    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee3457a-f692-40c5-8762-02f3141561ef" path="/var/lib/kubelet/pods/4ee3457a-f692-40c5-8762-02f3141561ef/volumes"
Oct 29 22:14:19.636496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.636473    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eecee06-ztv4z" event={"ID":"7fa68641-c5bc-46e1-8492-f46183338fe7","Type":"ContainerDied","Data":"96948f7dda3e459811bc6c4ddc5e8fdee0ef125dace1c61b097a08a3660b3ebb"}
Oct 29 22:14:19.636774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.636753    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96948f7dda3e459811bc6c4ddc5e8fdee0ef125dace1c61b097a08a3660b3ebb"
Oct 29 22:14:19.636896 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.636878    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98ad10af-94rzf" event={"ID":"59f43c8d-9dac-4f7d-94e1-e0100cd449d0","Type":"ContainerDied","Data":"8eb79e6364054338e06033dca5ed12cf4c70a08265c5fef61b60e17af1cf45af"}
Oct 29 22:14:19.636997 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.636984    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eb79e6364054338e06033dca5ed12cf4c70a08265c5fef61b60e17af1cf45af"
Oct 29 22:14:19.645042 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.645024    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"]
Oct 29 22:14:19.651402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.651257    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"]
Oct 29 22:14:19.651402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.651365    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"
Oct 29 22:14:19.660472 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.660416    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"]
Oct 29 22:14:19.665812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.665792    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"
Oct 29 22:14:19.668467 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.668449    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"]
Oct 29 22:14:19.681145 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.680983    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwsqk\" (UniqueName: \"kubernetes.io/projected/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-kube-api-access-cwsqk\") pod \"maas-api-revoke-sub-819d7332-65s7r\" (UID: \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"
Oct 29 22:14:19.681145 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.681059    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-819d7332-65s7r\" (UID: \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"
Oct 29 22:14:19.681331 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.681207    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eaa59b5-tzt5v\" (UID: \"2351c336-be09-42b5-a7d9-7b1806e4eccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"
Oct 29 22:14:19.681331 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.681264    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-819d7332-65s7r\" (UID: \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"
Oct 29 22:14:19.681331 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.681285    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2nnf\" (UniqueName: \"kubernetes.io/projected/2351c336-be09-42b5-a7d9-7b1806e4eccc-kube-api-access-j2nnf\") pod \"maas-api-revoke-sub-3eaa59b5-tzt5v\" (UID: \"2351c336-be09-42b5-a7d9-7b1806e4eccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"
Oct 29 22:14:19.681493 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.681374    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eaa59b5-tzt5v\" (UID: \"2351c336-be09-42b5-a7d9-7b1806e4eccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"
Oct 29 22:14:19.684248 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.684228    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"
Oct 29 22:14:19.712375 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.712348    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"]
Oct 29 22:14:19.784883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.784297    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsqk\" (UniqueName: \"kubernetes.io/projected/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-kube-api-access-cwsqk\") pod \"maas-api-revoke-sub-819d7332-65s7r\" (UID: \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"
Oct 29 22:14:19.784883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.784347    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-819d7332-65s7r\" (UID: \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"
Oct 29 22:14:19.784883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.784419    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wv62\" (UniqueName: \"kubernetes.io/projected/bda00a8c-7817-4294-902e-ba6098375eb3-kube-api-access-5wv62\") pod \"maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj\" (UID: \"bda00a8c-7817-4294-902e-ba6098375eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"
Oct 29 22:14:19.784883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.784440    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eaa59b5-tzt5v\" (UID: \"2351c336-be09-42b5-a7d9-7b1806e4eccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"
Oct 29 22:14:19.784883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.784488    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-819d7332-65s7r\" (UID: \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"
Oct 29 22:14:19.784883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.784508    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2nnf\" (UniqueName: \"kubernetes.io/projected/2351c336-be09-42b5-a7d9-7b1806e4eccc-kube-api-access-j2nnf\") pod \"maas-api-revoke-sub-3eaa59b5-tzt5v\" (UID: \"2351c336-be09-42b5-a7d9-7b1806e4eccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"
Oct 29 22:14:19.784883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.784546    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj\" (UID: \"bda00a8c-7817-4294-902e-ba6098375eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"
Oct 29 22:14:19.784883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.784603    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj\" (UID: \"bda00a8c-7817-4294-902e-ba6098375eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"
Oct 29 22:14:19.784883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.784654    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eaa59b5-tzt5v\" (UID: \"2351c336-be09-42b5-a7d9-7b1806e4eccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"
Oct 29 22:14:19.786967 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.786483    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-819d7332-65s7r\" (UID: \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"
Oct 29 22:14:19.788084 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.787794    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eaa59b5-tzt5v\" (UID: \"2351c336-be09-42b5-a7d9-7b1806e4eccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"
Oct 29 22:14:19.788410 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.788359    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eaa59b5-tzt5v\" (UID: \"2351c336-be09-42b5-a7d9-7b1806e4eccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"
Oct 29 22:14:19.791696 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.791606    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-819d7332-65s7r\" (UID: \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"
Oct 29 22:14:19.809715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.809696    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2nnf\" (UniqueName: \"kubernetes.io/projected/2351c336-be09-42b5-a7d9-7b1806e4eccc-kube-api-access-j2nnf\") pod \"maas-api-revoke-sub-3eaa59b5-tzt5v\" (UID: \"2351c336-be09-42b5-a7d9-7b1806e4eccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"
Oct 29 22:14:19.810417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.810398    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwsqk\" (UniqueName: \"kubernetes.io/projected/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-kube-api-access-cwsqk\") pod \"maas-api-revoke-sub-819d7332-65s7r\" (UID: \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"
Oct 29 22:14:19.885970 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.885938    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wv62\" (UniqueName: \"kubernetes.io/projected/bda00a8c-7817-4294-902e-ba6098375eb3-kube-api-access-5wv62\") pod \"maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj\" (UID: \"bda00a8c-7817-4294-902e-ba6098375eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"
Oct 29 22:14:19.886367 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.886047    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj\" (UID: \"bda00a8c-7817-4294-902e-ba6098375eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"
Oct 29 22:14:19.886367 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.886104    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj\" (UID: \"bda00a8c-7817-4294-902e-ba6098375eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"
Oct 29 22:14:19.886856 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.886833    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj\" (UID: \"bda00a8c-7817-4294-902e-ba6098375eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"
Oct 29 22:14:19.888443 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.888422    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj\" (UID: \"bda00a8c-7817-4294-902e-ba6098375eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"
Oct 29 22:14:19.908571 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.908543    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wv62\" (UniqueName: \"kubernetes.io/projected/bda00a8c-7817-4294-902e-ba6098375eb3-kube-api-access-5wv62\") pod \"maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj\" (UID: \"bda00a8c-7817-4294-902e-ba6098375eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"
Oct 29 22:14:19.943113 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.943053    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"
Oct 29 22:14:19.977164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.977134    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"
Oct 29 22:14:19.990742 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:19.990714    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"
Oct 29 22:14:20.035047 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:20.025580    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"]
Oct 29 22:14:20.040132 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:14:20.040095    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec21198_9a7b_44b1_af11_aa6a89c71c7c.slice/crio-3402cc1f1399bda1bdd6ffebe9acb5957b004ba4fc329e613c9127895fed9e27 WatchSource:0}: Error finding container 3402cc1f1399bda1bdd6ffebe9acb5957b004ba4fc329e613c9127895fed9e27: Status 404 returned error can't find the container with id 3402cc1f1399bda1bdd6ffebe9acb5957b004ba4fc329e613c9127895fed9e27
Oct 29 22:14:20.291338 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:20.291313    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"]
Oct 29 22:14:20.291609 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:14:20.291585    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2351c336_be09_42b5_a7d9_7b1806e4eccc.slice/crio-6b446dad08960a96f4e1e830237d29322726aa179e14a03db6f257c1adb56e52 WatchSource:0}: Error finding container 6b446dad08960a96f4e1e830237d29322726aa179e14a03db6f257c1adb56e52: Status 404 returned error can't find the container with id 6b446dad08960a96f4e1e830237d29322726aa179e14a03db6f257c1adb56e52
Oct 29 22:14:20.351610 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:20.351583    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"]
Oct 29 22:14:20.354860 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:14:20.354835    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda00a8c_7817_4294_902e_ba6098375eb3.slice/crio-bd1ea1bd54a8260eb0f02a79da3ba5b7893d2865f4b6e743161d69ca6d4d149c WatchSource:0}: Error finding container bd1ea1bd54a8260eb0f02a79da3ba5b7893d2865f4b6e743161d69ca6d4d149c: Status 404 returned error can't find the container with id bd1ea1bd54a8260eb0f02a79da3ba5b7893d2865f4b6e743161d69ca6d4d149c
Oct 29 22:14:20.523527 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:14:20.523499    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b4519a_2ca2_4da5_9b8b_44b3a8b4cf29.slice/crio-6c3f097045e9e8142a79a670b5e8840c40059a17f0ae1175d2926be752c550bb WatchSource:0}: Error finding container 6c3f097045e9e8142a79a670b5e8840c40059a17f0ae1175d2926be752c550bb: Status 404 returned error can't find the container with id 6c3f097045e9e8142a79a670b5e8840c40059a17f0ae1175d2926be752c550bb
Oct 29 22:14:20.525516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:20.525481    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"]
Oct 29 22:14:20.641496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:20.641454    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r" event={"ID":"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29","Type":"ContainerStarted","Data":"6c3f097045e9e8142a79a670b5e8840c40059a17f0ae1175d2926be752c550bb"}
Oct 29 22:14:20.643254 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:20.643228    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v" event={"ID":"2351c336-be09-42b5-a7d9-7b1806e4eccc","Type":"ContainerStarted","Data":"6b446dad08960a96f4e1e830237d29322726aa179e14a03db6f257c1adb56e52"}
Oct 29 22:14:20.646207 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:20.646182    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8" event={"ID":"3ec21198-9a7b-44b1-af11-aa6a89c71c7c","Type":"ContainerStarted","Data":"19e9dfb55baca2ba79f9f91b6cd93157835a95668d2031aa006ce6c26bcb284d"}
Oct 29 22:14:20.646320 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:20.646231    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8" event={"ID":"3ec21198-9a7b-44b1-af11-aa6a89c71c7c","Type":"ContainerStarted","Data":"3402cc1f1399bda1bdd6ffebe9acb5957b004ba4fc329e613c9127895fed9e27"}
Oct 29 22:14:20.647829 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:20.647802    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj" event={"ID":"bda00a8c-7817-4294-902e-ba6098375eb3","Type":"ContainerStarted","Data":"bd1ea1bd54a8260eb0f02a79da3ba5b7893d2865f4b6e743161d69ca6d4d149c"}
Oct 29 22:14:20.689397 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:20.689350    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8" podStartSLOduration=1.689334304 podStartE2EDuration="1.689334304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:20.687673954 +0000 UTC m=+1633.187716253" watchObservedRunningTime="2026-10-29 22:14:20.689334304 +0000 UTC m=+1633.189376597"
Oct 29 22:14:20.898745 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:20.898708    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr" podUID="e3a92df7-921d-4f5f-a18b-0c9494dfba37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:14:21.657781 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:21.657749    7082 generic.go:356] "Generic (PLEG): container finished" podID="2351c336-be09-42b5-a7d9-7b1806e4eccc" containerID="b3d4689be7ca5a0629cdc561b192fa2e5c8d687d8ee3a2aeb937defa195dac80" exitCode=0
Oct 29 22:14:21.658188 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:21.658164    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v" event={"ID":"2351c336-be09-42b5-a7d9-7b1806e4eccc","Type":"ContainerDied","Data":"b3d4689be7ca5a0629cdc561b192fa2e5c8d687d8ee3a2aeb937defa195dac80"}
Oct 29 22:14:21.659905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:21.659881    7082 generic.go:356] "Generic (PLEG): container finished" podID="3ec21198-9a7b-44b1-af11-aa6a89c71c7c" containerID="19e9dfb55baca2ba79f9f91b6cd93157835a95668d2031aa006ce6c26bcb284d" exitCode=0
Oct 29 22:14:21.660027 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:21.659962    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8" event={"ID":"3ec21198-9a7b-44b1-af11-aa6a89c71c7c","Type":"ContainerDied","Data":"19e9dfb55baca2ba79f9f91b6cd93157835a95668d2031aa006ce6c26bcb284d"}
Oct 29 22:14:21.661480 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:21.661454    7082 generic.go:356] "Generic (PLEG): container finished" podID="bda00a8c-7817-4294-902e-ba6098375eb3" containerID="991bd362b85d9b6c213886b30800f701810a602da8b3e47bfb4541b002e62025" exitCode=0
Oct 29 22:14:21.661589 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:21.661536    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj" event={"ID":"bda00a8c-7817-4294-902e-ba6098375eb3","Type":"ContainerDied","Data":"991bd362b85d9b6c213886b30800f701810a602da8b3e47bfb4541b002e62025"}
Oct 29 22:14:21.663237 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:21.663217    7082 generic.go:356] "Generic (PLEG): container finished" podID="d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29" containerID="974c5e1c74b82bd203a9602bedb4b727d126f691d38099b172f81ede56c68e3d" exitCode=0
Oct 29 22:14:21.663327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:21.663246    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r" event={"ID":"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29","Type":"ContainerDied","Data":"974c5e1c74b82bd203a9602bedb4b727d126f691d38099b172f81ede56c68e3d"}
Oct 29 22:14:23.044792 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.043281    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"
Oct 29 22:14:23.084818 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.083302    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda00a8c-7817-4294-902e-ba6098375eb3-kube-api-access-5wv62\" (UniqueName: \"kubernetes.io/projected/bda00a8c-7817-4294-902e-ba6098375eb3-kube-api-access-5wv62\") pod \"bda00a8c-7817-4294-902e-ba6098375eb3\" (UID: \"bda00a8c-7817-4294-902e-ba6098375eb3\") "
Oct 29 22:14:23.084818 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.083567    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-openshift-ca\") pod \"bda00a8c-7817-4294-902e-ba6098375eb3\" (UID: \"bda00a8c-7817-4294-902e-ba6098375eb3\") "
Oct 29 22:14:23.084818 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.083793    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-xks-ca\") pod \"bda00a8c-7817-4294-902e-ba6098375eb3\" (UID: \"bda00a8c-7817-4294-902e-ba6098375eb3\") "
Oct 29 22:14:23.086544 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.086521    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-openshift-ca" pod "bda00a8c-7817-4294-902e-ba6098375eb3" (UID: "bda00a8c-7817-4294-902e-ba6098375eb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:23.091007 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.090984    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda00a8c-7817-4294-902e-ba6098375eb3-kube-api-access-5wv62" pod "bda00a8c-7817-4294-902e-ba6098375eb3" (UID: "bda00a8c-7817-4294-902e-ba6098375eb3"). InnerVolumeSpecName "kube-api-access-5wv62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:23.104484 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.104457    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-xks-ca" pod "bda00a8c-7817-4294-902e-ba6098375eb3" (UID: "bda00a8c-7817-4294-902e-ba6098375eb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:23.142118 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.142018    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"
Oct 29 22:14:23.147831 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.147807    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"
Oct 29 22:14:23.152509 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.152485    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"
Oct 29 22:14:23.185797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.185726    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-openshift-ca\") pod \"2351c336-be09-42b5-a7d9-7b1806e4eccc\" (UID: \"2351c336-be09-42b5-a7d9-7b1806e4eccc\") "
Oct 29 22:14:23.185797 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.185794    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-xks-ca\") pod \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\" (UID: \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\") "
Oct 29 22:14:23.185988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.185931    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-xks-ca\") pod \"2351c336-be09-42b5-a7d9-7b1806e4eccc\" (UID: \"2351c336-be09-42b5-a7d9-7b1806e4eccc\") "
Oct 29 22:14:23.186055 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.186012    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-openshift-ca\") pod \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\" (UID: \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\") "
Oct 29 22:14:23.186444 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.186111    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-xks-ca\") pod \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\" (UID: \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\") "
Oct 29 22:14:23.186444 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.186180    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-kube-api-access-j9knk\" (UniqueName: \"kubernetes.io/projected/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-kube-api-access-j9knk\") pod \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\" (UID: \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\") "
Oct 29 22:14:23.186444 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.186210    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2351c336-be09-42b5-a7d9-7b1806e4eccc-kube-api-access-j2nnf\" (UniqueName: \"kubernetes.io/projected/2351c336-be09-42b5-a7d9-7b1806e4eccc-kube-api-access-j2nnf\") pod \"2351c336-be09-42b5-a7d9-7b1806e4eccc\" (UID: \"2351c336-be09-42b5-a7d9-7b1806e4eccc\") "
Oct 29 22:14:23.186444 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.186289    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-kube-api-access-cwsqk\" (UniqueName: \"kubernetes.io/projected/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-kube-api-access-cwsqk\") pod \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\" (UID: \"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29\") "
Oct 29 22:14:23.186444 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.186372    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-openshift-ca\") pod \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\" (UID: \"3ec21198-9a7b-44b1-af11-aa6a89c71c7c\") "
Oct 29 22:14:23.187126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.187097    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-openshift-ca" pod "2351c336-be09-42b5-a7d9-7b1806e4eccc" (UID: "2351c336-be09-42b5-a7d9-7b1806e4eccc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:23.187126 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.187117    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:23.187317 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.187138    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bda00a8c-7817-4294-902e-ba6098375eb3-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:23.187317 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.187152    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wv62\" (UniqueName: \"kubernetes.io/projected/bda00a8c-7817-4294-902e-ba6098375eb3-kube-api-access-5wv62\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:23.187490 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.187465    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-openshift-ca" pod "3ec21198-9a7b-44b1-af11-aa6a89c71c7c" (UID: "3ec21198-9a7b-44b1-af11-aa6a89c71c7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:23.188801 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.188775    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-openshift-ca" pod "d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29" (UID: "d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:23.189942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.189904    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-xks-ca" pod "d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29" (UID: "d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:23.191855 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.191830    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-kube-api-access-j9knk" pod "3ec21198-9a7b-44b1-af11-aa6a89c71c7c" (UID: "3ec21198-9a7b-44b1-af11-aa6a89c71c7c"). InnerVolumeSpecName "kube-api-access-j9knk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:23.192147 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.192125    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-xks-ca" pod "3ec21198-9a7b-44b1-af11-aa6a89c71c7c" (UID: "3ec21198-9a7b-44b1-af11-aa6a89c71c7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:23.192384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.192365    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-kube-api-access-cwsqk" pod "d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29" (UID: "d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29"). InnerVolumeSpecName "kube-api-access-cwsqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:23.194219 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.194186    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-xks-ca" pod "2351c336-be09-42b5-a7d9-7b1806e4eccc" (UID: "2351c336-be09-42b5-a7d9-7b1806e4eccc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:23.194219 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.194192    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2351c336-be09-42b5-a7d9-7b1806e4eccc-kube-api-access-j2nnf" pod "2351c336-be09-42b5-a7d9-7b1806e4eccc" (UID: "2351c336-be09-42b5-a7d9-7b1806e4eccc"). InnerVolumeSpecName "kube-api-access-j2nnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:23.288885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.288841    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:23.288885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.288879    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:23.289141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.288896    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:23.289141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.288909    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9knk\" (UniqueName: \"kubernetes.io/projected/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-kube-api-access-j9knk\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:23.289141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.288922    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2nnf\" (UniqueName: \"kubernetes.io/projected/2351c336-be09-42b5-a7d9-7b1806e4eccc-kube-api-access-j2nnf\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:23.289141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.288934    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwsqk\" (UniqueName: \"kubernetes.io/projected/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-kube-api-access-cwsqk\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:23.289141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.288946    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ec21198-9a7b-44b1-af11-aa6a89c71c7c-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:23.289141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.288959    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2351c336-be09-42b5-a7d9-7b1806e4eccc-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:23.289141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.288973    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:23.695881 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.695850    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v"
Oct 29 22:14:23.696056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.695835    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eaa59b5-tzt5v" event={"ID":"2351c336-be09-42b5-a7d9-7b1806e4eccc","Type":"ContainerDied","Data":"6b446dad08960a96f4e1e830237d29322726aa179e14a03db6f257c1adb56e52"}
Oct 29 22:14:23.696135 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.696082    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b446dad08960a96f4e1e830237d29322726aa179e14a03db6f257c1adb56e52"
Oct 29 22:14:23.697697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.697680    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"
Oct 29 22:14:23.698092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.697686    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8" event={"ID":"3ec21198-9a7b-44b1-af11-aa6a89c71c7c","Type":"ContainerDied","Data":"3402cc1f1399bda1bdd6ffebe9acb5957b004ba4fc329e613c9127895fed9e27"}
Oct 29 22:14:23.698222 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.698117    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3402cc1f1399bda1bdd6ffebe9acb5957b004ba4fc329e613c9127895fed9e27"
Oct 29 22:14:23.699845 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.699826    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"
Oct 29 22:14:23.700275 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.700252    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj" event={"ID":"bda00a8c-7817-4294-902e-ba6098375eb3","Type":"ContainerDied","Data":"bd1ea1bd54a8260eb0f02a79da3ba5b7893d2865f4b6e743161d69ca6d4d149c"}
Oct 29 22:14:23.700418 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.700295    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd1ea1bd54a8260eb0f02a79da3ba5b7893d2865f4b6e743161d69ca6d4d149c"
Oct 29 22:14:23.702250 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.702230    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r" event={"ID":"d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29","Type":"ContainerDied","Data":"6c3f097045e9e8142a79a670b5e8840c40059a17f0ae1175d2926be752c550bb"}
Oct 29 22:14:23.702389 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.702373    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c3f097045e9e8142a79a670b5e8840c40059a17f0ae1175d2926be752c550bb"
Oct 29 22:14:23.702389 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:23.702248    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-819d7332-65s7r"
Oct 29 22:14:24.305888 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.305860    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:14:24.419359 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.419323    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-data\") pod \"bf767f5a-7583-4f85-a50a-70c504a28898\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") "
Oct 29 22:14:24.419560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.419415    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf767f5a-7583-4f85-a50a-70c504a28898-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf767f5a-7583-4f85-a50a-70c504a28898-istiod-ca-cert\") pod \"bf767f5a-7583-4f85-a50a-70c504a28898\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") "
Oct 29 22:14:24.419560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.419540    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-kube-api-access-gn727\" (UniqueName: \"kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-kube-api-access-gn727\") pod \"bf767f5a-7583-4f85-a50a-70c504a28898\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") "
Oct 29 22:14:24.419770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.419669    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-credential-socket\") pod \"bf767f5a-7583-4f85-a50a-70c504a28898\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") "
Oct 29 22:14:24.419770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.419662    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-data" pod "bf767f5a-7583-4f85-a50a-70c504a28898" (UID: "bf767f5a-7583-4f85-a50a-70c504a28898"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:24.419770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.419726    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-istio-token\" (UniqueName: \"kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-istio-token\") pod \"bf767f5a-7583-4f85-a50a-70c504a28898\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") "
Oct 29 22:14:24.419942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.419864    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-socket\") pod \"bf767f5a-7583-4f85-a50a-70c504a28898\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") "
Oct 29 22:14:24.419942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.419897    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-envoy\") pod \"bf767f5a-7583-4f85-a50a-70c504a28898\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") "
Oct 29 22:14:24.419942 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.419931    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-credential-socket" pod "bf767f5a-7583-4f85-a50a-70c504a28898" (UID: "bf767f5a-7583-4f85-a50a-70c504a28898"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:24.420128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.419943    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf767f5a-7583-4f85-a50a-70c504a28898-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf767f5a-7583-4f85-a50a-70c504a28898-istio-podinfo\") pod \"bf767f5a-7583-4f85-a50a-70c504a28898\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") "
Oct 29 22:14:24.420128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.419952    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf767f5a-7583-4f85-a50a-70c504a28898-istiod-ca-cert" pod "bf767f5a-7583-4f85-a50a-70c504a28898" (UID: "bf767f5a-7583-4f85-a50a-70c504a28898"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:24.420128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.420006    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-certs\") pod \"bf767f5a-7583-4f85-a50a-70c504a28898\" (UID: \"bf767f5a-7583-4f85-a50a-70c504a28898\") "
Oct 29 22:14:24.420302 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.420137    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-socket" pod "bf767f5a-7583-4f85-a50a-70c504a28898" (UID: "bf767f5a-7583-4f85-a50a-70c504a28898"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:24.420753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.420723    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-certs" pod "bf767f5a-7583-4f85-a50a-70c504a28898" (UID: "bf767f5a-7583-4f85-a50a-70c504a28898"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:24.420991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.420802    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf767f5a-7583-4f85-a50a-70c504a28898-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.420991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.420823    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.420991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.420838    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.420991 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.420851    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.422399 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.422376    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf767f5a-7583-4f85-a50a-70c504a28898-istio-podinfo" pod "bf767f5a-7583-4f85-a50a-70c504a28898" (UID: "bf767f5a-7583-4f85-a50a-70c504a28898"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:24.422502 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.422478    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-istio-token" pod "bf767f5a-7583-4f85-a50a-70c504a28898" (UID: "bf767f5a-7583-4f85-a50a-70c504a28898"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:24.422675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.422615    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-kube-api-access-gn727" pod "bf767f5a-7583-4f85-a50a-70c504a28898" (UID: "bf767f5a-7583-4f85-a50a-70c504a28898"). InnerVolumeSpecName "kube-api-access-gn727". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:24.422779 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.422761    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-envoy" pod "bf767f5a-7583-4f85-a50a-70c504a28898" (UID: "bf767f5a-7583-4f85-a50a-70c504a28898"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:24.522968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.522890    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.522968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.522925    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.522968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.522939    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf767f5a-7583-4f85-a50a-70c504a28898-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.522968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.522952    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf767f5a-7583-4f85-a50a-70c504a28898-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.522968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.522965    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn727\" (UniqueName: \"kubernetes.io/projected/bf767f5a-7583-4f85-a50a-70c504a28898-kube-api-access-gn727\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.714700 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.714665    7082 generic.go:356] "Generic (PLEG): container finished" podID="bf767f5a-7583-4f85-a50a-70c504a28898" containerID="d0d60ef12968e8110a371e797d6943db767c0e8451bea3dfbfa06998a851c6ee" exitCode=0
Oct 29 22:14:24.714903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.714712    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"
Oct 29 22:14:24.714903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.714753    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" event={"ID":"bf767f5a-7583-4f85-a50a-70c504a28898","Type":"ContainerDied","Data":"d0d60ef12968e8110a371e797d6943db767c0e8451bea3dfbfa06998a851c6ee"}
Oct 29 22:14:24.714903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.714818    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298" event={"ID":"bf767f5a-7583-4f85-a50a-70c504a28898","Type":"ContainerDied","Data":"e4aa5531b14c60dabf7d9c0af4cd7dcb2353ea2b8123f4e0725edb05b80251e0"}
Oct 29 22:14:24.714903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.714844    7082 scope.go:122] "RemoveContainer" containerID="d0d60ef12968e8110a371e797d6943db767c0e8451bea3dfbfa06998a851c6ee"
Oct 29 22:14:24.733710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.733687    7082 scope.go:122] "RemoveContainer" containerID="a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973"
Oct 29 22:14:24.745403 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.745377    7082 scope.go:122] "RemoveContainer" containerID="d0d60ef12968e8110a371e797d6943db767c0e8451bea3dfbfa06998a851c6ee"
Oct 29 22:14:24.745687 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:24.745664    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d60ef12968e8110a371e797d6943db767c0e8451bea3dfbfa06998a851c6ee\": container with ID starting with d0d60ef12968e8110a371e797d6943db767c0e8451bea3dfbfa06998a851c6ee not found: ID does not exist" containerID="d0d60ef12968e8110a371e797d6943db767c0e8451bea3dfbfa06998a851c6ee"
Oct 29 22:14:24.745739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.745698    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d60ef12968e8110a371e797d6943db767c0e8451bea3dfbfa06998a851c6ee"} err="failed to get container status \"d0d60ef12968e8110a371e797d6943db767c0e8451bea3dfbfa06998a851c6ee\": rpc error: code = NotFound desc = could not find container \"d0d60ef12968e8110a371e797d6943db767c0e8451bea3dfbfa06998a851c6ee\": container with ID starting with d0d60ef12968e8110a371e797d6943db767c0e8451bea3dfbfa06998a851c6ee not found: ID does not exist"
Oct 29 22:14:24.745739 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.745715    7082 scope.go:122] "RemoveContainer" containerID="a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973"
Oct 29 22:14:24.745929 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:24.745913    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973\": container with ID starting with a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973 not found: ID does not exist" containerID="a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973"
Oct 29 22:14:24.745968 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.745936    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973"} err="failed to get container status \"a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973\": rpc error: code = NotFound desc = could not find container \"a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973\": container with ID starting with a5e32880500422d1af9ad22e494e9d81e548f427c71c7dc60e03daf891216973 not found: ID does not exist"
Oct 29 22:14:24.770924 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.770892    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"]
Oct 29 22:14:24.776141 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:24.776080    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-pv298"]
Oct 29 22:14:25.282978 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:25.282945    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"]
Oct 29 22:14:25.283272 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:25.283240    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" podUID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" containerName="istio-proxy" containerID="cri-o://19713b4347b065627314564bc48679a01f81b7613b28b3ba6564fce40621b315" gracePeriod=30
Oct 29 22:14:25.289092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:25.289065    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"]
Oct 29 22:14:25.289288 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:25.289264    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" containerID="cri-o://7809d0bf655c2143cfaa371b2b79452a5eded8cc81c5f1a54f295a560aeb575e" gracePeriod=30
Oct 29 22:14:25.303905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:25.302440    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"]
Oct 29 22:14:25.303905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:25.302691    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" containerID="cri-o://2a3b4996408a46bec23a8d3928e7dfb372c2008bb905f64c527637a4d748279f" gracePeriod=30
Oct 29 22:14:25.663665 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:25.660546    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" path="/var/lib/kubelet/pods/bf767f5a-7583-4f85-a50a-70c504a28898/volumes"
Oct 29 22:14:26.980662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:26.977960    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:26.980662 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:26.978029    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:27.055001 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:27.049598    7082 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:27.055001 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:27.049672    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:28.414287 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:28.414246    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:14:28.904926 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:28.904394    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk" podUID="b742a630-e75b-4040-8521-12388f3d0427" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:14:29.061694 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:29.061239    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"]
Oct 29 22:14:29.085673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:29.081926    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a940bc-tk65p"]
Oct 29 22:14:29.635350 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:29.635314    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8065e1cd-dfe3-4a1e-83f0-58d970f03ff9" path="/var/lib/kubelet/pods/8065e1cd-dfe3-4a1e-83f0-58d970f03ff9/volumes"
Oct 29 22:14:30.528451 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.528103    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 22:14:30.573434 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.573399    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-socket\") pod \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") "
Oct 29 22:14:30.573769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.573469    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-token\" (UniqueName: \"kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-token\") pod \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") "
Oct 29 22:14:30.573769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.573553    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-kube-api-access-qgl5b\" (UniqueName: \"kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-kube-api-access-qgl5b\") pod \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") "
Oct 29 22:14:30.573769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.573613    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-envoy\") pod \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") "
Oct 29 22:14:30.573769 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.573729    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-podinfo\") pod \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") "
Oct 29 22:14:30.574201 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.573795    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-data\") pod \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") "
Oct 29 22:14:30.574201 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.573904    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-credential-socket\") pod \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") "
Oct 29 22:14:30.574201 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.574028    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istiod-ca-cert\") pod \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") "
Oct 29 22:14:30.574201 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.574094    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-certs\") pod \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\" (UID: \"d56d6c21-ed20-4380-b2f5-69b9d168f5c5\") "
Oct 29 22:14:30.575225 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.575178    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-certs" pod "d56d6c21-ed20-4380-b2f5-69b9d168f5c5" (UID: "d56d6c21-ed20-4380-b2f5-69b9d168f5c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.575812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.575777    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-data" pod "d56d6c21-ed20-4380-b2f5-69b9d168f5c5" (UID: "d56d6c21-ed20-4380-b2f5-69b9d168f5c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.576034 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.576001    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-credential-socket" pod "d56d6c21-ed20-4380-b2f5-69b9d168f5c5" (UID: "d56d6c21-ed20-4380-b2f5-69b9d168f5c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.576409 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.576389    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-socket" pod "d56d6c21-ed20-4380-b2f5-69b9d168f5c5" (UID: "d56d6c21-ed20-4380-b2f5-69b9d168f5c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.576620 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.576568    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istiod-ca-cert" pod "d56d6c21-ed20-4380-b2f5-69b9d168f5c5" (UID: "d56d6c21-ed20-4380-b2f5-69b9d168f5c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:30.584645 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.584579    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-envoy" pod "d56d6c21-ed20-4380-b2f5-69b9d168f5c5" (UID: "d56d6c21-ed20-4380-b2f5-69b9d168f5c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.591915 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.591887    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-podinfo" pod "d56d6c21-ed20-4380-b2f5-69b9d168f5c5" (UID: "d56d6c21-ed20-4380-b2f5-69b9d168f5c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:30.592974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.592938    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-kube-api-access-qgl5b" pod "d56d6c21-ed20-4380-b2f5-69b9d168f5c5" (UID: "d56d6c21-ed20-4380-b2f5-69b9d168f5c5"). InnerVolumeSpecName "kube-api-access-qgl5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.598165 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.598139    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-token" pod "d56d6c21-ed20-4380-b2f5-69b9d168f5c5" (UID: "d56d6c21-ed20-4380-b2f5-69b9d168f5c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.643157 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.643130    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:14:30.677708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.677675    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84310bdd-05f6-4107-9430-68fecce3fcec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84310bdd-05f6-4107-9430-68fecce3fcec-istiod-ca-cert\") pod \"84310bdd-05f6-4107-9430-68fecce3fcec\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") "
Oct 29 22:14:30.677851 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.677787    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-certs\") pod \"84310bdd-05f6-4107-9430-68fecce3fcec\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") "
Oct 29 22:14:30.677933 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.677872    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-data\") pod \"84310bdd-05f6-4107-9430-68fecce3fcec\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") "
Oct 29 22:14:30.677992 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.677941    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-kube-api-access-g4zb5\" (UniqueName: \"kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-kube-api-access-g4zb5\") pod \"84310bdd-05f6-4107-9430-68fecce3fcec\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") "
Oct 29 22:14:30.678104 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.678016    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-credential-socket\") pod \"84310bdd-05f6-4107-9430-68fecce3fcec\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") "
Oct 29 22:14:30.678104 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.678089    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84310bdd-05f6-4107-9430-68fecce3fcec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84310bdd-05f6-4107-9430-68fecce3fcec-istio-podinfo\") pod \"84310bdd-05f6-4107-9430-68fecce3fcec\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") "
Oct 29 22:14:30.678206 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.678115    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-envoy\") pod \"84310bdd-05f6-4107-9430-68fecce3fcec\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") "
Oct 29 22:14:30.678206 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.678175    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-istio-token\" (UniqueName: \"kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-istio-token\") pod \"84310bdd-05f6-4107-9430-68fecce3fcec\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") "
Oct 29 22:14:30.678317 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.678252    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-socket\") pod \"84310bdd-05f6-4107-9430-68fecce3fcec\" (UID: \"84310bdd-05f6-4107-9430-68fecce3fcec\") "
Oct 29 22:14:30.680559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.678745    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-credential-socket" pod "84310bdd-05f6-4107-9430-68fecce3fcec" (UID: "84310bdd-05f6-4107-9430-68fecce3fcec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.680559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.679171    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.680559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.679192    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.680559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.679209    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.680559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.679226    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.680559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.679238    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.680559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.679252    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgl5b\" (UniqueName: \"kubernetes.io/projected/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-kube-api-access-qgl5b\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.680559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.679265    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.680559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.679278    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.680559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.679290    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.680559 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.679302    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56d6c21-ed20-4380-b2f5-69b9d168f5c5-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.683774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.681530    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-socket" pod "84310bdd-05f6-4107-9430-68fecce3fcec" (UID: "84310bdd-05f6-4107-9430-68fecce3fcec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.683774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.681678    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-data" pod "84310bdd-05f6-4107-9430-68fecce3fcec" (UID: "84310bdd-05f6-4107-9430-68fecce3fcec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.683774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.681733    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-certs" pod "84310bdd-05f6-4107-9430-68fecce3fcec" (UID: "84310bdd-05f6-4107-9430-68fecce3fcec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.683774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.681876    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84310bdd-05f6-4107-9430-68fecce3fcec-istiod-ca-cert" pod "84310bdd-05f6-4107-9430-68fecce3fcec" (UID: "84310bdd-05f6-4107-9430-68fecce3fcec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:30.683774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.683539    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-istio-token" pod "84310bdd-05f6-4107-9430-68fecce3fcec" (UID: "84310bdd-05f6-4107-9430-68fecce3fcec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.684457 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.684429    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-envoy" pod "84310bdd-05f6-4107-9430-68fecce3fcec" (UID: "84310bdd-05f6-4107-9430-68fecce3fcec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.684457 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.684443    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-kube-api-access-g4zb5" pod "84310bdd-05f6-4107-9430-68fecce3fcec" (UID: "84310bdd-05f6-4107-9430-68fecce3fcec"). InnerVolumeSpecName "kube-api-access-g4zb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.685432 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.685409    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84310bdd-05f6-4107-9430-68fecce3fcec-istio-podinfo" pod "84310bdd-05f6-4107-9430-68fecce3fcec" (UID: "84310bdd-05f6-4107-9430-68fecce3fcec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:30.687858 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.687781    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 22:14:30.780722 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.780688    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-certs\") pod \"c530e17d-eb72-4b77-b2e6-669d8225b75a\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") "
Oct 29 22:14:30.780902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.780761    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-credential-socket\") pod \"c530e17d-eb72-4b77-b2e6-669d8225b75a\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") "
Oct 29 22:14:30.780902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.780796    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-token\" (UniqueName: \"kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-token\") pod \"c530e17d-eb72-4b77-b2e6-669d8225b75a\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") "
Oct 29 22:14:30.780902 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.780868    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-podinfo\") pod \"c530e17d-eb72-4b77-b2e6-669d8225b75a\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") "
Oct 29 22:14:30.781094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.780905    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-envoy\") pod \"c530e17d-eb72-4b77-b2e6-669d8225b75a\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") "
Oct 29 22:14:30.781094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.781036    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-data\") pod \"c530e17d-eb72-4b77-b2e6-669d8225b75a\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") "
Oct 29 22:14:30.781094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.781082    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-credential-socket" pod "c530e17d-eb72-4b77-b2e6-669d8225b75a" (UID: "c530e17d-eb72-4b77-b2e6-669d8225b75a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.781265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.781095    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c530e17d-eb72-4b77-b2e6-669d8225b75a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c530e17d-eb72-4b77-b2e6-669d8225b75a-istiod-ca-cert\") pod \"c530e17d-eb72-4b77-b2e6-669d8225b75a\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") "
Oct 29 22:14:30.781265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.781170    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-kube-api-access-wvlmt\" (UniqueName: \"kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-kube-api-access-wvlmt\") pod \"c530e17d-eb72-4b77-b2e6-669d8225b75a\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") "
Oct 29 22:14:30.781265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.781250    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-socket\") pod \"c530e17d-eb72-4b77-b2e6-669d8225b75a\" (UID: \"c530e17d-eb72-4b77-b2e6-669d8225b75a\") "
Oct 29 22:14:30.781614 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.781569    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-certs" pod "c530e17d-eb72-4b77-b2e6-669d8225b75a" (UID: "c530e17d-eb72-4b77-b2e6-669d8225b75a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.782117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.781848    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-data" pod "c530e17d-eb72-4b77-b2e6-669d8225b75a" (UID: "c530e17d-eb72-4b77-b2e6-669d8225b75a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.782362 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782053    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-socket" pod "c530e17d-eb72-4b77-b2e6-669d8225b75a" (UID: "c530e17d-eb72-4b77-b2e6-669d8225b75a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.782446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782095    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c530e17d-eb72-4b77-b2e6-669d8225b75a-istiod-ca-cert" pod "c530e17d-eb72-4b77-b2e6-669d8225b75a" (UID: "c530e17d-eb72-4b77-b2e6-669d8225b75a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:30.782606 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782586    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84310bdd-05f6-4107-9430-68fecce3fcec-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.782791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782610    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.782791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782645    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.782791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782659    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c530e17d-eb72-4b77-b2e6-669d8225b75a-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.782791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782670    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.782791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782683    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4zb5\" (UniqueName: \"kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-kube-api-access-g4zb5\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.782791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782696    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.782791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782709    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84310bdd-05f6-4107-9430-68fecce3fcec-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.782791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782722    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.782791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782734    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84310bdd-05f6-4107-9430-68fecce3fcec-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.782791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782746    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84310bdd-05f6-4107-9430-68fecce3fcec-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.782791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782758    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.782791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.782772    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.785957 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.785055    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-podinfo" pod "c530e17d-eb72-4b77-b2e6-669d8225b75a" (UID: "c530e17d-eb72-4b77-b2e6-669d8225b75a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:30.786724 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.786702    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-token" pod "c530e17d-eb72-4b77-b2e6-669d8225b75a" (UID: "c530e17d-eb72-4b77-b2e6-669d8225b75a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.786805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.786749    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-kube-api-access-wvlmt" pod "c530e17d-eb72-4b77-b2e6-669d8225b75a" (UID: "c530e17d-eb72-4b77-b2e6-669d8225b75a"). InnerVolumeSpecName "kube-api-access-wvlmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.787772 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.787747    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-envoy" pod "c530e17d-eb72-4b77-b2e6-669d8225b75a" (UID: "c530e17d-eb72-4b77-b2e6-669d8225b75a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.823130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.823101    7082 generic.go:356] "Generic (PLEG): container finished" podID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" containerID="19713b4347b065627314564bc48679a01f81b7613b28b3ba6564fce40621b315" exitCode=0
Oct 29 22:14:30.823808 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.823165    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"
Oct 29 22:14:30.823808 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.823194    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" event={"ID":"d56d6c21-ed20-4380-b2f5-69b9d168f5c5","Type":"ContainerDied","Data":"19713b4347b065627314564bc48679a01f81b7613b28b3ba6564fce40621b315"}
Oct 29 22:14:30.823808 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.823270    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z" event={"ID":"d56d6c21-ed20-4380-b2f5-69b9d168f5c5","Type":"ContainerDied","Data":"6c83f432e75164f44f15ecf407d534fa9bfbd8d6a0e8ef0ea80ce8ce8b9e3dc4"}
Oct 29 22:14:30.823808 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.823305    7082 scope.go:122] "RemoveContainer" containerID="19713b4347b065627314564bc48679a01f81b7613b28b3ba6564fce40621b315"
Oct 29 22:14:30.824926 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.824907    7082 generic.go:356] "Generic (PLEG): container finished" podID="84310bdd-05f6-4107-9430-68fecce3fcec" containerID="7809d0bf655c2143cfaa371b2b79452a5eded8cc81c5f1a54f295a560aeb575e" exitCode=0
Oct 29 22:14:30.825059 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.825033    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" event={"ID":"84310bdd-05f6-4107-9430-68fecce3fcec","Type":"ContainerDied","Data":"7809d0bf655c2143cfaa371b2b79452a5eded8cc81c5f1a54f295a560aeb575e"}
Oct 29 22:14:30.825213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.825081    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw" event={"ID":"84310bdd-05f6-4107-9430-68fecce3fcec","Type":"ContainerDied","Data":"a5121ad22562f5911dd37c45fbdffde1d7f2506a5fae99c45dc062dedf1ed408"}
Oct 29 22:14:30.825213 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.825152    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"
Oct 29 22:14:30.829898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.829670    7082 generic.go:356] "Generic (PLEG): container finished" podID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerID="2a3b4996408a46bec23a8d3928e7dfb372c2008bb905f64c527637a4d748279f" exitCode=0
Oct 29 22:14:30.829898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.829730    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"
Oct 29 22:14:30.829898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.829770    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" event={"ID":"c530e17d-eb72-4b77-b2e6-669d8225b75a","Type":"ContainerDied","Data":"2a3b4996408a46bec23a8d3928e7dfb372c2008bb905f64c527637a4d748279f"}
Oct 29 22:14:30.829898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.829845    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9" event={"ID":"c530e17d-eb72-4b77-b2e6-669d8225b75a","Type":"ContainerDied","Data":"641b82ca2ac59d23def76979c48ad47a10e06d62e34c742e41ac908b1a597a4f"}
Oct 29 22:14:30.842827 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.842803    7082 scope.go:122] "RemoveContainer" containerID="19713b4347b065627314564bc48679a01f81b7613b28b3ba6564fce40621b315"
Oct 29 22:14:30.843309 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:30.843277    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19713b4347b065627314564bc48679a01f81b7613b28b3ba6564fce40621b315\": container with ID starting with 19713b4347b065627314564bc48679a01f81b7613b28b3ba6564fce40621b315 not found: ID does not exist" containerID="19713b4347b065627314564bc48679a01f81b7613b28b3ba6564fce40621b315"
Oct 29 22:14:30.843414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.843356    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19713b4347b065627314564bc48679a01f81b7613b28b3ba6564fce40621b315"} err="failed to get container status \"19713b4347b065627314564bc48679a01f81b7613b28b3ba6564fce40621b315\": rpc error: code = NotFound desc = could not find container \"19713b4347b065627314564bc48679a01f81b7613b28b3ba6564fce40621b315\": container with ID starting with 19713b4347b065627314564bc48679a01f81b7613b28b3ba6564fce40621b315 not found: ID does not exist"
Oct 29 22:14:30.843414 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.843381    7082 scope.go:122] "RemoveContainer" containerID="7809d0bf655c2143cfaa371b2b79452a5eded8cc81c5f1a54f295a560aeb575e"
Oct 29 22:14:30.857241 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.857217    7082 scope.go:122] "RemoveContainer" containerID="7809d0bf655c2143cfaa371b2b79452a5eded8cc81c5f1a54f295a560aeb575e"
Oct 29 22:14:30.857578 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:30.857554    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7809d0bf655c2143cfaa371b2b79452a5eded8cc81c5f1a54f295a560aeb575e\": container with ID starting with 7809d0bf655c2143cfaa371b2b79452a5eded8cc81c5f1a54f295a560aeb575e not found: ID does not exist" containerID="7809d0bf655c2143cfaa371b2b79452a5eded8cc81c5f1a54f295a560aeb575e"
Oct 29 22:14:30.857817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.857589    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7809d0bf655c2143cfaa371b2b79452a5eded8cc81c5f1a54f295a560aeb575e"} err="failed to get container status \"7809d0bf655c2143cfaa371b2b79452a5eded8cc81c5f1a54f295a560aeb575e\": rpc error: code = NotFound desc = could not find container \"7809d0bf655c2143cfaa371b2b79452a5eded8cc81c5f1a54f295a560aeb575e\": container with ID starting with 7809d0bf655c2143cfaa371b2b79452a5eded8cc81c5f1a54f295a560aeb575e not found: ID does not exist"
Oct 29 22:14:30.857817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.857612    7082 scope.go:122] "RemoveContainer" containerID="2a3b4996408a46bec23a8d3928e7dfb372c2008bb905f64c527637a4d748279f"
Oct 29 22:14:30.869042 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.869020    7082 scope.go:122] "RemoveContainer" containerID="2a3b4996408a46bec23a8d3928e7dfb372c2008bb905f64c527637a4d748279f"
Oct 29 22:14:30.869364 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:30.869345    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a3b4996408a46bec23a8d3928e7dfb372c2008bb905f64c527637a4d748279f\": container with ID starting with 2a3b4996408a46bec23a8d3928e7dfb372c2008bb905f64c527637a4d748279f not found: ID does not exist" containerID="2a3b4996408a46bec23a8d3928e7dfb372c2008bb905f64c527637a4d748279f"
Oct 29 22:14:30.869478 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.869369    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a3b4996408a46bec23a8d3928e7dfb372c2008bb905f64c527637a4d748279f"} err="failed to get container status \"2a3b4996408a46bec23a8d3928e7dfb372c2008bb905f64c527637a4d748279f\": rpc error: code = NotFound desc = could not find container \"2a3b4996408a46bec23a8d3928e7dfb372c2008bb905f64c527637a4d748279f\": container with ID starting with 2a3b4996408a46bec23a8d3928e7dfb372c2008bb905f64c527637a4d748279f not found: ID does not exist"
Oct 29 22:14:30.882098 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.882064    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"]
Oct 29 22:14:30.882221 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.882105    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-6295z"]
Oct 29 22:14:30.884263 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.884240    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.884263 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.884262    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.884438 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.884272    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvlmt\" (UniqueName: \"kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-kube-api-access-wvlmt\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.884438 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.884281    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c530e17d-eb72-4b77-b2e6-669d8225b75a-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.897166 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.897143    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr" podUID="e3a92df7-921d-4f5f-a18b-0c9494dfba37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:14:30.945409 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.945380    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"]
Oct 29 22:14:30.949567 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.949548    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-dlpd9"]
Oct 29 22:14:30.981663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.981619    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"]
Oct 29 22:14:30.988726 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:30.988697    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wvqnw"]
Oct 29 22:14:31.701350 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.701307    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" path="/var/lib/kubelet/pods/84310bdd-05f6-4107-9430-68fecce3fcec/volumes"
Oct 29 22:14:31.704040 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.704017    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" path="/var/lib/kubelet/pods/c530e17d-eb72-4b77-b2e6-669d8225b75a/volumes"
Oct 29 22:14:31.707050 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.707026    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" path="/var/lib/kubelet/pods/d56d6c21-ed20-4380-b2f5-69b9d168f5c5/volumes"
Oct 29 22:14:31.732066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.732038    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"
Oct 29 22:14:31.804932 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.804399    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3a92df7-921d-4f5f-a18b-0c9494dfba37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a92df7-921d-4f5f-a18b-0c9494dfba37-plugins-config-volume\") pod \"e3a92df7-921d-4f5f-a18b-0c9494dfba37\" (UID: \"e3a92df7-921d-4f5f-a18b-0c9494dfba37\") "
Oct 29 22:14:31.804932 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.804672    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3a92df7-921d-4f5f-a18b-0c9494dfba37-kube-api-access-j8bqw\" (UniqueName: \"kubernetes.io/projected/e3a92df7-921d-4f5f-a18b-0c9494dfba37-kube-api-access-j8bqw\") pod \"e3a92df7-921d-4f5f-a18b-0c9494dfba37\" (UID: \"e3a92df7-921d-4f5f-a18b-0c9494dfba37\") "
Oct 29 22:14:31.813095 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.813046    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a92df7-921d-4f5f-a18b-0c9494dfba37-plugins-config-volume" pod "e3a92df7-921d-4f5f-a18b-0c9494dfba37" (UID: "e3a92df7-921d-4f5f-a18b-0c9494dfba37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:31.816056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.816022    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a92df7-921d-4f5f-a18b-0c9494dfba37-kube-api-access-j8bqw" pod "e3a92df7-921d-4f5f-a18b-0c9494dfba37" (UID: "e3a92df7-921d-4f5f-a18b-0c9494dfba37"). InnerVolumeSpecName "kube-api-access-j8bqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:31.831599 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.831531    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a92df7-921d-4f5f-a18b-0c9494dfba37-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:31.831599 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.831565    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8bqw\" (UniqueName: \"kubernetes.io/projected/e3a92df7-921d-4f5f-a18b-0c9494dfba37-kube-api-access-j8bqw\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:31.847283 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.846825    7082 generic.go:356] "Generic (PLEG): container finished" podID="b742a630-e75b-4040-8521-12388f3d0427" containerID="18cc190494e31d0dfacc0b5498f1f7da571342517d08184baeae71bd2d5883d8" exitCode=0
Oct 29 22:14:31.847283 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.846968    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk" event={"ID":"b742a630-e75b-4040-8521-12388f3d0427","Type":"ContainerDied","Data":"18cc190494e31d0dfacc0b5498f1f7da571342517d08184baeae71bd2d5883d8"}
Oct 29 22:14:31.849821 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.849802    7082 generic.go:356] "Generic (PLEG): container finished" podID="e3a92df7-921d-4f5f-a18b-0c9494dfba37" containerID="2182d653f74f40e443dd8498d2133d0df2fbf2720f27132ca808221758b13d17" exitCode=0
Oct 29 22:14:31.850060 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.849966    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr" event={"ID":"e3a92df7-921d-4f5f-a18b-0c9494dfba37","Type":"ContainerDied","Data":"2182d653f74f40e443dd8498d2133d0df2fbf2720f27132ca808221758b13d17"}
Oct 29 22:14:31.851725 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.849987    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"
Oct 29 22:14:31.855176 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.851863    7082 scope.go:122] "RemoveContainer" containerID="2182d653f74f40e443dd8498d2133d0df2fbf2720f27132ca808221758b13d17"
Oct 29 22:14:31.855176 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.850606    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr" event={"ID":"e3a92df7-921d-4f5f-a18b-0c9494dfba37","Type":"ContainerDied","Data":"167020242e1f156f5983b9e693e836bc32a1885a21b3368138943503962ff728"}
Oct 29 22:14:31.880171 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.880148    7082 scope.go:122] "RemoveContainer" containerID="2182d653f74f40e443dd8498d2133d0df2fbf2720f27132ca808221758b13d17"
Oct 29 22:14:31.880277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.880171    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"
Oct 29 22:14:31.880740 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:31.880716    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2182d653f74f40e443dd8498d2133d0df2fbf2720f27132ca808221758b13d17\": container with ID starting with 2182d653f74f40e443dd8498d2133d0df2fbf2720f27132ca808221758b13d17 not found: ID does not exist" containerID="2182d653f74f40e443dd8498d2133d0df2fbf2720f27132ca808221758b13d17"
Oct 29 22:14:31.880840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.880745    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2182d653f74f40e443dd8498d2133d0df2fbf2720f27132ca808221758b13d17"} err="failed to get container status \"2182d653f74f40e443dd8498d2133d0df2fbf2720f27132ca808221758b13d17\": rpc error: code = NotFound desc = could not find container \"2182d653f74f40e443dd8498d2133d0df2fbf2720f27132ca808221758b13d17\": container with ID starting with 2182d653f74f40e443dd8498d2133d0df2fbf2720f27132ca808221758b13d17 not found: ID does not exist"
Oct 29 22:14:31.907964 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.907935    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"]
Oct 29 22:14:31.911703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:31.911683    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-743a5e-d4b59856b-xhqfr"]
Oct 29 22:14:32.034390 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:32.034354    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b742a630-e75b-4040-8521-12388f3d0427-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b742a630-e75b-4040-8521-12388f3d0427-plugins-config-volume\") pod \"b742a630-e75b-4040-8521-12388f3d0427\" (UID: \"b742a630-e75b-4040-8521-12388f3d0427\") "
Oct 29 22:14:32.034561 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:32.034434    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b742a630-e75b-4040-8521-12388f3d0427-kube-api-access-dh7jw\" (UniqueName: \"kubernetes.io/projected/b742a630-e75b-4040-8521-12388f3d0427-kube-api-access-dh7jw\") pod \"b742a630-e75b-4040-8521-12388f3d0427\" (UID: \"b742a630-e75b-4040-8521-12388f3d0427\") "
Oct 29 22:14:32.034868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:32.034843    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b742a630-e75b-4040-8521-12388f3d0427-plugins-config-volume" pod "b742a630-e75b-4040-8521-12388f3d0427" (UID: "b742a630-e75b-4040-8521-12388f3d0427"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:32.035138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:32.035119    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b742a630-e75b-4040-8521-12388f3d0427-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:32.036787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:32.036762    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b742a630-e75b-4040-8521-12388f3d0427-kube-api-access-dh7jw" pod "b742a630-e75b-4040-8521-12388f3d0427" (UID: "b742a630-e75b-4040-8521-12388f3d0427"). InnerVolumeSpecName "kube-api-access-dh7jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:32.136762 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:32.136692    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh7jw\" (UniqueName: \"kubernetes.io/projected/b742a630-e75b-4040-8521-12388f3d0427-kube-api-access-dh7jw\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:32.865527 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:32.865498    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"
Oct 29 22:14:32.866014 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:32.865540    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk" event={"ID":"b742a630-e75b-4040-8521-12388f3d0427","Type":"ContainerDied","Data":"9a567e48f4f160a6490945767b0b753d54473348af7fa2c46d2cd72076a814a9"}
Oct 29 22:14:32.866014 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:32.865608    7082 scope.go:122] "RemoveContainer" containerID="18cc190494e31d0dfacc0b5498f1f7da571342517d08184baeae71bd2d5883d8"
Oct 29 22:14:32.917311 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:32.917284    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"]
Oct 29 22:14:32.932601 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:32.932562    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-743a5e-8db88755-cc8rk"]
Oct 29 22:14:33.613529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:33.613494    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"]
Oct 29 22:14:33.614026 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:33.613996    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9" podUID="4e480bf6-b96a-4315-be7e-9180c8173fab" containerName="payload-processing" containerID="cri-o://49c4029e0ce261218c8b80ccfdf869d1fc8243770bdceec6965b8582d98509b2" gracePeriod=30
Oct 29 22:14:33.744112 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:33.744062    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b742a630-e75b-4040-8521-12388f3d0427" path="/var/lib/kubelet/pods/b742a630-e75b-4040-8521-12388f3d0427/volumes"
Oct 29 22:14:33.745027 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:33.744988    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3a92df7-921d-4f5f-a18b-0c9494dfba37" path="/var/lib/kubelet/pods/e3a92df7-921d-4f5f-a18b-0c9494dfba37/volumes"
Oct 29 22:14:33.745688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:33.745670    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"]
Oct 29 22:14:33.746044 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:33.746019    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8" podUID="d570bb54-a40a-44f7-afe5-875bc98185b6" containerName="payload-pre-processing" containerID="cri-o://90768ac43e208e9becc0e83931dc40e98394b37ffc3c73dc57ac9d44475478f0" gracePeriod=30
Oct 29 22:14:33.833091 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:33.833055    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"]
Oct 29 22:14:33.844982 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:33.844948    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-743a5e-84fs4"]
Oct 29 22:14:33.881432 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:33.881395    7082 generic.go:356] "Generic (PLEG): container finished" podID="d570bb54-a40a-44f7-afe5-875bc98185b6" containerID="90768ac43e208e9becc0e83931dc40e98394b37ffc3c73dc57ac9d44475478f0" exitCode=0
Oct 29 22:14:33.881870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:33.881619    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8" event={"ID":"d570bb54-a40a-44f7-afe5-875bc98185b6","Type":"ContainerDied","Data":"90768ac43e208e9becc0e83931dc40e98394b37ffc3c73dc57ac9d44475478f0"}
Oct 29 22:14:33.913649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:33.912797    7082 generic.go:356] "Generic (PLEG): container finished" podID="4e480bf6-b96a-4315-be7e-9180c8173fab" containerID="49c4029e0ce261218c8b80ccfdf869d1fc8243770bdceec6965b8582d98509b2" exitCode=0
Oct 29 22:14:33.913649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:33.912910    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9" event={"ID":"4e480bf6-b96a-4315-be7e-9180c8173fab","Type":"ContainerDied","Data":"49c4029e0ce261218c8b80ccfdf869d1fc8243770bdceec6965b8582d98509b2"}
Oct 29 22:14:34.185192 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.185155    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"]
Oct 29 22:14:34.186703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.186609    7082 cpu_manager.go:414] "containerMap: removing container" podUID="2351c336-be09-42b5-a7d9-7b1806e4eccc" containerName="revoke-keys"
Oct 29 22:14:34.186858 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.186843    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2351c336-be09-42b5-a7d9-7b1806e4eccc" containerName="revoke-keys"
Oct 29 22:14:34.186953 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.186944    7082 cpu_manager.go:414] "containerMap: removing container" podUID="bda00a8c-7817-4294-902e-ba6098375eb3" containerName="revoke-keys"
Oct 29 22:14:34.187047 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.187035    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda00a8c-7817-4294-902e-ba6098375eb3" containerName="revoke-keys"
Oct 29 22:14:34.187170 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.187157    7082 cpu_manager.go:414] "containerMap: removing container" podUID="b742a630-e75b-4040-8521-12388f3d0427" containerName="payload-pre-processing"
Oct 29 22:14:34.187251 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.187240    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b742a630-e75b-4040-8521-12388f3d0427" containerName="payload-pre-processing"
Oct 29 22:14:34.187564 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.187344    7082 cpu_manager.go:414] "containerMap: removing container" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy"
Oct 29 22:14:34.188711 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.188691    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy"
Oct 29 22:14:34.188835 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.188824    7082 cpu_manager.go:414] "containerMap: removing container" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy"
Oct 29 22:14:34.188901 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.188892    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy"
Oct 29 22:14:34.188975 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.188967    7082 cpu_manager.go:414] "containerMap: removing container" podUID="3ec21198-9a7b-44b1-af11-aa6a89c71c7c" containerName="revoke-keys"
Oct 29 22:14:34.189034 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.189025    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec21198-9a7b-44b1-af11-aa6a89c71c7c" containerName="revoke-keys"
Oct 29 22:14:34.189100 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.189092    7082 cpu_manager.go:414] "containerMap: removing container" podUID="d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29" containerName="revoke-keys"
Oct 29 22:14:34.189157 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.189149    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29" containerName="revoke-keys"
Oct 29 22:14:34.189218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.189211    7082 cpu_manager.go:414] "containerMap: removing container" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy"
Oct 29 22:14:34.189282 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.189274    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy"
Oct 29 22:14:34.189360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.189352    7082 cpu_manager.go:414] "containerMap: removing container" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy"
Oct 29 22:14:34.189417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.189409    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy"
Oct 29 22:14:34.189532 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.189525    7082 cpu_manager.go:414] "containerMap: removing container" podUID="e3a92df7-921d-4f5f-a18b-0c9494dfba37" containerName="payload-processing"
Oct 29 22:14:34.189595 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.189587    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a92df7-921d-4f5f-a18b-0c9494dfba37" containerName="payload-processing"
Oct 29 22:14:34.189689 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.189679    7082 cpu_manager.go:414] "containerMap: removing container" podUID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" containerName="istio-proxy"
Oct 29 22:14:34.189774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.189764    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" containerName="istio-proxy"
Oct 29 22:14:34.189996 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.189986    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="84310bdd-05f6-4107-9430-68fecce3fcec" containerName="istio-proxy"
Oct 29 22:14:34.190082 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.190073    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a92df7-921d-4f5f-a18b-0c9494dfba37" containerName="payload-processing"
Oct 29 22:14:34.190158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.190150    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56d6c21-ed20-4380-b2f5-69b9d168f5c5" containerName="istio-proxy"
Oct 29 22:14:34.190234 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.190225    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="2351c336-be09-42b5-a7d9-7b1806e4eccc" containerName="revoke-keys"
Oct 29 22:14:34.190328 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.190319    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda00a8c-7817-4294-902e-ba6098375eb3" containerName="revoke-keys"
Oct 29 22:14:34.190398 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.190390    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="b742a630-e75b-4040-8521-12388f3d0427" containerName="payload-pre-processing"
Oct 29 22:14:34.190473 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.190464    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="c530e17d-eb72-4b77-b2e6-669d8225b75a" containerName="istio-proxy"
Oct 29 22:14:34.190550 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.190541    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy"
Oct 29 22:14:34.190757 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.190744    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec21198-9a7b-44b1-af11-aa6a89c71c7c" containerName="revoke-keys"
Oct 29 22:14:34.190857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.190848    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29" containerName="revoke-keys"
Oct 29 22:14:34.190931 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.190922    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf767f5a-7583-4f85-a50a-70c504a28898" containerName="istio-proxy"
Oct 29 22:14:34.245525 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.240949    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"
Oct 29 22:14:34.245525 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.242509    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"
Oct 29 22:14:34.251717 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.251686    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"]
Oct 29 22:14:34.260272 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.260226    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:14:34.260650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.260609    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:14:34.267018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.266992    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:14:34.267118 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.267072    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:14:34.270437 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.270411    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn8hg\" (UniqueName: \"kubernetes.io/projected/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-kube-api-access-dn8hg\") pod \"maas-api-revoke-sub-767ac17e-rlcls\" (UID: \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"
Oct 29 22:14:34.270713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.270620    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767ac17e-rlcls\" (UID: \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"
Oct 29 22:14:34.270713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.270706    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767ac17e-rlcls\" (UID: \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"
Oct 29 22:14:34.371782 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.371753    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e480bf6-b96a-4315-be7e-9180c8173fab-kube-api-access-2nfx9\" (UniqueName: \"kubernetes.io/projected/4e480bf6-b96a-4315-be7e-9180c8173fab-kube-api-access-2nfx9\") pod \"4e480bf6-b96a-4315-be7e-9180c8173fab\" (UID: \"4e480bf6-b96a-4315-be7e-9180c8173fab\") "
Oct 29 22:14:34.371949 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.371802    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e480bf6-b96a-4315-be7e-9180c8173fab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e480bf6-b96a-4315-be7e-9180c8173fab-plugins-config-volume\") pod \"4e480bf6-b96a-4315-be7e-9180c8173fab\" (UID: \"4e480bf6-b96a-4315-be7e-9180c8173fab\") "
Oct 29 22:14:34.372210 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.372185    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e480bf6-b96a-4315-be7e-9180c8173fab-plugins-config-volume" pod "4e480bf6-b96a-4315-be7e-9180c8173fab" (UID: "4e480bf6-b96a-4315-be7e-9180c8173fab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:34.372210 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.372195    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn8hg\" (UniqueName: \"kubernetes.io/projected/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-kube-api-access-dn8hg\") pod \"maas-api-revoke-sub-767ac17e-rlcls\" (UID: \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"
Oct 29 22:14:34.372448 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.372427    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767ac17e-rlcls\" (UID: \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"
Oct 29 22:14:34.372534 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.372496    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767ac17e-rlcls\" (UID: \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"
Oct 29 22:14:34.372655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.372617    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e480bf6-b96a-4315-be7e-9180c8173fab-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:34.373325 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.373251    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767ac17e-rlcls\" (UID: \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"
Oct 29 22:14:34.374171 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.374151    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e480bf6-b96a-4315-be7e-9180c8173fab-kube-api-access-2nfx9" pod "4e480bf6-b96a-4315-be7e-9180c8173fab" (UID: "4e480bf6-b96a-4315-be7e-9180c8173fab"). InnerVolumeSpecName "kube-api-access-2nfx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:34.375067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.375049    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767ac17e-rlcls\" (UID: \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"
Oct 29 22:14:34.382909 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.382891    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"
Oct 29 22:14:34.394421 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.394395    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn8hg\" (UniqueName: \"kubernetes.io/projected/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-kube-api-access-dn8hg\") pod \"maas-api-revoke-sub-767ac17e-rlcls\" (UID: \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"
Oct 29 22:14:34.473257 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.473160    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d570bb54-a40a-44f7-afe5-875bc98185b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d570bb54-a40a-44f7-afe5-875bc98185b6-plugins-config-volume\") pod \"d570bb54-a40a-44f7-afe5-875bc98185b6\" (UID: \"d570bb54-a40a-44f7-afe5-875bc98185b6\") "
Oct 29 22:14:34.474105 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.474081    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d570bb54-a40a-44f7-afe5-875bc98185b6-kube-api-access-z6xfz\" (UniqueName: \"kubernetes.io/projected/d570bb54-a40a-44f7-afe5-875bc98185b6-kube-api-access-z6xfz\") pod \"d570bb54-a40a-44f7-afe5-875bc98185b6\" (UID: \"d570bb54-a40a-44f7-afe5-875bc98185b6\") "
Oct 29 22:14:34.475587 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.475558    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nfx9\" (UniqueName: \"kubernetes.io/projected/4e480bf6-b96a-4315-be7e-9180c8173fab-kube-api-access-2nfx9\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:34.477489 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.473912    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d570bb54-a40a-44f7-afe5-875bc98185b6-plugins-config-volume" pod "d570bb54-a40a-44f7-afe5-875bc98185b6" (UID: "d570bb54-a40a-44f7-afe5-875bc98185b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:34.482730 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.482694    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"]
Oct 29 22:14:34.487248 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.487200    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d570bb54-a40a-44f7-afe5-875bc98185b6-kube-api-access-z6xfz" pod "d570bb54-a40a-44f7-afe5-875bc98185b6" (UID: "d570bb54-a40a-44f7-afe5-875bc98185b6"). InnerVolumeSpecName "kube-api-access-z6xfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:34.488715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.488688    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-75db5b-hz8nj"]
Oct 29 22:14:34.567034 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.566997    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"
Oct 29 22:14:34.579064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.579040    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d570bb54-a40a-44f7-afe5-875bc98185b6-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:34.579191 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.579068    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6xfz\" (UniqueName: \"kubernetes.io/projected/d570bb54-a40a-44f7-afe5-875bc98185b6-kube-api-access-z6xfz\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:34.880650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.880605    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"]
Oct 29 22:14:34.880807 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:14:34.880684    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a8d55f_8e42_4f30_8f53_0fa9af1b2cde.slice/crio-7b3907af92549d6d772cbb509d0c13531e16a0da7b4b99dda77038dcfadfe91c WatchSource:0}: Error finding container 7b3907af92549d6d772cbb509d0c13531e16a0da7b4b99dda77038dcfadfe91c: Status 404 returned error can't find the container with id 7b3907af92549d6d772cbb509d0c13531e16a0da7b4b99dda77038dcfadfe91c
Oct 29 22:14:34.931617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.931418    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8" event={"ID":"d570bb54-a40a-44f7-afe5-875bc98185b6","Type":"ContainerDied","Data":"a0bf79270d25fde12de765eed9921fe904d3550319ab75ec2ae95d906e602998"}
Oct 29 22:14:34.931617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.931524    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"
Oct 29 22:14:34.931617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.931535    7082 scope.go:122] "RemoveContainer" containerID="90768ac43e208e9becc0e83931dc40e98394b37ffc3c73dc57ac9d44475478f0"
Oct 29 22:14:34.934718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.933577    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9" event={"ID":"4e480bf6-b96a-4315-be7e-9180c8173fab","Type":"ContainerDied","Data":"ad30c3cbf3be60e9dde51b7430460bf52bbfc2639bcac1b76e9ee0b688ee7054"}
Oct 29 22:14:34.934718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.933729    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"
Oct 29 22:14:34.936529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.936500    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls" event={"ID":"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde","Type":"ContainerStarted","Data":"7b3907af92549d6d772cbb509d0c13531e16a0da7b4b99dda77038dcfadfe91c"}
Oct 29 22:14:34.947070 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.947052    7082 scope.go:122] "RemoveContainer" containerID="49c4029e0ce261218c8b80ccfdf869d1fc8243770bdceec6965b8582d98509b2"
Oct 29 22:14:34.997995 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:34.997926    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"]
Oct 29 22:14:35.005307 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:35.005284    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-75db5b-7fc7f76d46-97vb9"]
Oct 29 22:14:35.036143 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:35.036121    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"]
Oct 29 22:14:35.042552 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:35.042531    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-9fc6687d7-jstq8"]
Oct 29 22:14:35.633543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:35.633509    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20783eec-14f3-4dd3-ada8-cd52fcbe6f98" path="/var/lib/kubelet/pods/20783eec-14f3-4dd3-ada8-cd52fcbe6f98/volumes"
Oct 29 22:14:35.634164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:35.634135    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e480bf6-b96a-4315-be7e-9180c8173fab" path="/var/lib/kubelet/pods/4e480bf6-b96a-4315-be7e-9180c8173fab/volumes"
Oct 29 22:14:35.634761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:35.634738    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda00a8c-7817-4294-902e-ba6098375eb3" path="/var/lib/kubelet/pods/bda00a8c-7817-4294-902e-ba6098375eb3/volumes"
Oct 29 22:14:35.635783 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:35.635762    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d570bb54-a40a-44f7-afe5-875bc98185b6" path="/var/lib/kubelet/pods/d570bb54-a40a-44f7-afe5-875bc98185b6/volumes"
Oct 29 22:14:35.946394 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:35.946318    7082 generic.go:356] "Generic (PLEG): container finished" podID="86a8d55f-8e42-4f30-8f53-0fa9af1b2cde" containerID="f11e28feeba55301a63d2e5032dfbeeeceba7c97fb47065771eb0de244e4d2c6" exitCode=0
Oct 29 22:14:35.946822 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:35.946406    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls" event={"ID":"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde","Type":"ContainerDied","Data":"f11e28feeba55301a63d2e5032dfbeeeceba7c97fb47065771eb0de244e4d2c6"}
Oct 29 22:14:37.165498 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:37.165472    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"
Oct 29 22:14:37.252712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:37.250550    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-kube-api-access-dn8hg\" (UniqueName: \"kubernetes.io/projected/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-kube-api-access-dn8hg\") pod \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\" (UID: \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\") "
Oct 29 22:14:37.252712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:37.250807    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-xks-ca\") pod \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\" (UID: \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\") "
Oct 29 22:14:37.252712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:37.250873    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-openshift-ca\") pod \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\" (UID: \"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde\") "
Oct 29 22:14:37.252712 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:37.251886    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-openshift-ca" pod "86a8d55f-8e42-4f30-8f53-0fa9af1b2cde" (UID: "86a8d55f-8e42-4f30-8f53-0fa9af1b2cde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:37.275284 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:37.275223    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-kube-api-access-dn8hg" pod "86a8d55f-8e42-4f30-8f53-0fa9af1b2cde" (UID: "86a8d55f-8e42-4f30-8f53-0fa9af1b2cde"). InnerVolumeSpecName "kube-api-access-dn8hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:37.302265 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:37.296980    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-xks-ca" pod "86a8d55f-8e42-4f30-8f53-0fa9af1b2cde" (UID: "86a8d55f-8e42-4f30-8f53-0fa9af1b2cde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:37.353258 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:37.353229    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn8hg\" (UniqueName: \"kubernetes.io/projected/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-kube-api-access-dn8hg\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:37.353258 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:37.353256    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:37.353456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:37.353268    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a8d55f-8e42-4f30-8f53-0fa9af1b2cde-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:38.121110 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:38.121072    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls" event={"ID":"86a8d55f-8e42-4f30-8f53-0fa9af1b2cde","Type":"ContainerDied","Data":"7b3907af92549d6d772cbb509d0c13531e16a0da7b4b99dda77038dcfadfe91c"}
Oct 29 22:14:38.121410 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:38.121393    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b3907af92549d6d772cbb509d0c13531e16a0da7b4b99dda77038dcfadfe91c"
Oct 29 22:14:38.121581 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:38.121501    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767ac17e-rlcls"
Oct 29 22:14:38.440364 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:38.440159    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"]
Oct 29 22:14:38.440886 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:38.440444    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy" containerID="cri-o://99eaa8dd4a77b1b75062415752e29ee672ad20cb2cdbcf1c6bd9bb3bdf2085b5" gracePeriod=30
Oct 29 22:14:38.459731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:38.459699    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"]
Oct 29 22:14:38.460001 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:38.459954    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy" containerID="cri-o://f2ec93756b3a9e6b49a393771834c3f05bb2c5c12e7817d9a7b0a94a48869c8a" gracePeriod=30
Oct 29 22:14:38.464978 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:38.464947    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"]
Oct 29 22:14:38.465208 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:38.465185    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy" containerID="cri-o://e8f11674dc4876ad34473b84353bcee1d2af4b7ad0641a8074cb593b5ec91c45" gracePeriod=30
Oct 29 22:14:38.469400 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:38.469375    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"]
Oct 29 22:14:38.469614 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:38.469571    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy" containerID="cri-o://8ce199d2fcbefb880766e3756239e96330e08912f3b858f9ab8d7a9b8d706065" gracePeriod=30
Oct 29 22:14:41.491579 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:41.491543    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"]
Oct 29 22:14:41.492140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:41.491810    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerName="istio-proxy" containerID="cri-o://d1676651cfffbb392920180808eb75a168a8783429a5c2e286d541bbdaf6e009" gracePeriod=30
Oct 29 22:14:41.511617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:41.510600    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"]
Oct 29 22:14:41.511617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:41.510832    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy" containerID="cri-o://75abdf7b32b7a08079db296a280754d8aca7410046713628c21a0189290f5d38" gracePeriod=30
Oct 29 22:14:41.520105 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:41.520072    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"]
Oct 29 22:14:41.520340 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:41.520298    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy" containerID="cri-o://b7f92f156ecb8f1145be58a99a7e5443b9f50a5c0386018c29ce83fd21ccb0ad" gracePeriod=30
Oct 29 22:14:42.445044 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:42.443427    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"]
Oct 29 22:14:42.445044 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:42.443747    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz" podUID="027d1e1a-2888-45b7-b47e-2b62ca191f7a" containerName="istio-proxy" containerID="cri-o://32814cff4a4183d3ce8a8114c365e76cdefaeb5dbad2d2d7190c63cbc07fa82a" gracePeriod=30
Oct 29 22:14:42.455691 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:42.455660    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"]
Oct 29 22:14:42.455928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:42.455882    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb" podUID="bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" containerName="istio-proxy" containerID="cri-o://6ff3769da58d33e970c6e2f32744eb238e97eebefee07dfe880d9b0f660a61f3" gracePeriod=30
Oct 29 22:14:43.158927 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.158885    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"]
Oct 29 22:14:43.159733 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.159710    7082 cpu_manager.go:414] "containerMap: removing container" podUID="86a8d55f-8e42-4f30-8f53-0fa9af1b2cde" containerName="revoke-keys"
Oct 29 22:14:43.159733 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.159731    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a8d55f-8e42-4f30-8f53-0fa9af1b2cde" containerName="revoke-keys"
Oct 29 22:14:43.159923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.159771    7082 cpu_manager.go:414] "containerMap: removing container" podUID="4e480bf6-b96a-4315-be7e-9180c8173fab" containerName="payload-processing"
Oct 29 22:14:43.159923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.159779    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e480bf6-b96a-4315-be7e-9180c8173fab" containerName="payload-processing"
Oct 29 22:14:43.159923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.159800    7082 cpu_manager.go:414] "containerMap: removing container" podUID="d570bb54-a40a-44f7-afe5-875bc98185b6" containerName="payload-pre-processing"
Oct 29 22:14:43.159923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.159808    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d570bb54-a40a-44f7-afe5-875bc98185b6" containerName="payload-pre-processing"
Oct 29 22:14:43.160159 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.159989    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e480bf6-b96a-4315-be7e-9180c8173fab" containerName="payload-processing"
Oct 29 22:14:43.160159 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.160005    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="d570bb54-a40a-44f7-afe5-875bc98185b6" containerName="payload-pre-processing"
Oct 29 22:14:43.160159 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.160026    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a8d55f-8e42-4f30-8f53-0fa9af1b2cde" containerName="revoke-keys"
Oct 29 22:14:43.164147 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.164122    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"
Oct 29 22:14:43.185085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.185060    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"]
Oct 29 22:14:43.191243 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.191221    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:14:43.191422 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.191395    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:14:43.191422 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.191272    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:14:43.191648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.191540    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:14:43.209094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.207330    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj\" (UID: \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"
Oct 29 22:14:43.209094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.207389    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppqpj\" (UniqueName: \"kubernetes.io/projected/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-kube-api-access-ppqpj\") pod \"maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj\" (UID: \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"
Oct 29 22:14:43.209094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.207583    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj\" (UID: \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"
Oct 29 22:14:43.309541 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.309511    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj\" (UID: \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"
Oct 29 22:14:43.309541 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.309552    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppqpj\" (UniqueName: \"kubernetes.io/projected/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-kube-api-access-ppqpj\") pod \"maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj\" (UID: \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"
Oct 29 22:14:43.309791 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.309777    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj\" (UID: \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"
Oct 29 22:14:43.310353 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.310325    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj\" (UID: \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"
Oct 29 22:14:43.312018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.311997    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj\" (UID: \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"
Oct 29 22:14:43.334301 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.334280    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppqpj\" (UniqueName: \"kubernetes.io/projected/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-kube-api-access-ppqpj\") pod \"maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj\" (UID: \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"
Oct 29 22:14:43.460002 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.459923    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:43.460002 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.459988    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:43.479284 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.479250    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"
Oct 29 22:14:43.802997 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.802966    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"]
Oct 29 22:14:43.816742 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:14:43.816711    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69cbc6f_03a9_4469_bfcb_718296f3f8bd.slice/crio-64bc2a0428746fd8e91620d627e1a4bb11e5365237c95515bb3eb037d6fab3c0 WatchSource:0}: Error finding container 64bc2a0428746fd8e91620d627e1a4bb11e5365237c95515bb3eb037d6fab3c0: Status 404 returned error can't find the container with id 64bc2a0428746fd8e91620d627e1a4bb11e5365237c95515bb3eb037d6fab3c0
Oct 29 22:14:43.836541 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.836517    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 22:14:43.924106 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.923759    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-data\") pod \"f50a168b-99ca-411d-b32a-19021ca6e029\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") "
Oct 29 22:14:43.924106 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.924059    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-data" pod "f50a168b-99ca-411d-b32a-19021ca6e029" (UID: "f50a168b-99ca-411d-b32a-19021ca6e029"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.927817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.924400    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-envoy\") pod \"f50a168b-99ca-411d-b32a-19021ca6e029\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") "
Oct 29 22:14:43.927817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.924527    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f50a168b-99ca-411d-b32a-19021ca6e029-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50a168b-99ca-411d-b32a-19021ca6e029-istio-podinfo\") pod \"f50a168b-99ca-411d-b32a-19021ca6e029\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") "
Oct 29 22:14:43.927817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.924596    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-certs\") pod \"f50a168b-99ca-411d-b32a-19021ca6e029\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") "
Oct 29 22:14:43.927817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.924687    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-credential-socket\") pod \"f50a168b-99ca-411d-b32a-19021ca6e029\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") "
Oct 29 22:14:43.927817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.924752    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-kube-api-access-hskn5\" (UniqueName: \"kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-kube-api-access-hskn5\") pod \"f50a168b-99ca-411d-b32a-19021ca6e029\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") "
Oct 29 22:14:43.927817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.924817    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-socket\") pod \"f50a168b-99ca-411d-b32a-19021ca6e029\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") "
Oct 29 22:14:43.927817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.924878    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f50a168b-99ca-411d-b32a-19021ca6e029-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50a168b-99ca-411d-b32a-19021ca6e029-istiod-ca-cert\") pod \"f50a168b-99ca-411d-b32a-19021ca6e029\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") "
Oct 29 22:14:43.927817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.924920    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-istio-token\" (UniqueName: \"kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-istio-token\") pod \"f50a168b-99ca-411d-b32a-19021ca6e029\" (UID: \"f50a168b-99ca-411d-b32a-19021ca6e029\") "
Oct 29 22:14:43.927817 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.925608    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.928908 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.926946    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f50a168b-99ca-411d-b32a-19021ca6e029-istiod-ca-cert" pod "f50a168b-99ca-411d-b32a-19021ca6e029" (UID: "f50a168b-99ca-411d-b32a-19021ca6e029"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:43.928908 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.927672    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-socket" pod "f50a168b-99ca-411d-b32a-19021ca6e029" (UID: "f50a168b-99ca-411d-b32a-19021ca6e029"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.928908 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.927736    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-certs" pod "f50a168b-99ca-411d-b32a-19021ca6e029" (UID: "f50a168b-99ca-411d-b32a-19021ca6e029"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.929167 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.927790    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-credential-socket" pod "f50a168b-99ca-411d-b32a-19021ca6e029" (UID: "f50a168b-99ca-411d-b32a-19021ca6e029"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.936442 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.935140    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f50a168b-99ca-411d-b32a-19021ca6e029-istio-podinfo" pod "f50a168b-99ca-411d-b32a-19021ca6e029" (UID: "f50a168b-99ca-411d-b32a-19021ca6e029"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:43.938981 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.938922    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-envoy" pod "f50a168b-99ca-411d-b32a-19021ca6e029" (UID: "f50a168b-99ca-411d-b32a-19021ca6e029"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.939754 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.939574    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-kube-api-access-hskn5" pod "f50a168b-99ca-411d-b32a-19021ca6e029" (UID: "f50a168b-99ca-411d-b32a-19021ca6e029"). InnerVolumeSpecName "kube-api-access-hskn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.941676 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.940996    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-istio-token" pod "f50a168b-99ca-411d-b32a-19021ca6e029" (UID: "f50a168b-99ca-411d-b32a-19021ca6e029"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.945746 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.945711    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 22:14:43.955534 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.955514    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 22:14:43.958698 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:43.958604    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 22:14:44.030173 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030092    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-socket\") pod \"84119759-f427-4a8d-960e-8b342fca0d38\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") "
Oct 29 22:14:44.030173 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030134    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-podinfo\") pod \"3969f40e-b7c2-4839-85ef-998ab9d08122\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") "
Oct 29 22:14:44.030173 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030159    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-envoy\") pod \"84119759-f427-4a8d-960e-8b342fca0d38\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") "
Oct 29 22:14:44.030453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030182    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istiod-ca-cert\") pod \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") "
Oct 29 22:14:44.030453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030201    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-certs\") pod \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") "
Oct 29 22:14:44.030453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030217    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-data\") pod \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") "
Oct 29 22:14:44.030453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030245    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-envoy\") pod \"3969f40e-b7c2-4839-85ef-998ab9d08122\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") "
Oct 29 22:14:44.030453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030263    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-credential-socket\") pod \"3969f40e-b7c2-4839-85ef-998ab9d08122\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") "
Oct 29 22:14:44.030453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030278    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-istio-token\" (UniqueName: \"kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-istio-token\") pod \"84119759-f427-4a8d-960e-8b342fca0d38\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") "
Oct 29 22:14:44.030453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030300    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-kube-api-access-m94hd\" (UniqueName: \"kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-kube-api-access-m94hd\") pod \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") "
Oct 29 22:14:44.030453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030332    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3969f40e-b7c2-4839-85ef-998ab9d08122-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3969f40e-b7c2-4839-85ef-998ab9d08122-istiod-ca-cert\") pod \"3969f40e-b7c2-4839-85ef-998ab9d08122\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") "
Oct 29 22:14:44.030453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030364    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-kube-api-access-sq7z2\" (UniqueName: \"kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-kube-api-access-sq7z2\") pod \"3969f40e-b7c2-4839-85ef-998ab9d08122\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") "
Oct 29 22:14:44.030453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030384    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-data\") pod \"3969f40e-b7c2-4839-85ef-998ab9d08122\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") "
Oct 29 22:14:44.030453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030411    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84119759-f427-4a8d-960e-8b342fca0d38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84119759-f427-4a8d-960e-8b342fca0d38-istio-podinfo\") pod \"84119759-f427-4a8d-960e-8b342fca0d38\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") "
Oct 29 22:14:44.030453 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030447    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-credential-socket\") pod \"84119759-f427-4a8d-960e-8b342fca0d38\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") "
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030477    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-certs\") pod \"3969f40e-b7c2-4839-85ef-998ab9d08122\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") "
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030499    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-token\" (UniqueName: \"kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-token\") pod \"3969f40e-b7c2-4839-85ef-998ab9d08122\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") "
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030542    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-podinfo\") pod \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") "
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030574    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-kube-api-access-r5pd9\" (UniqueName: \"kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-kube-api-access-r5pd9\") pod \"84119759-f427-4a8d-960e-8b342fca0d38\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") "
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030654    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-socket\") pod \"3969f40e-b7c2-4839-85ef-998ab9d08122\" (UID: \"3969f40e-b7c2-4839-85ef-998ab9d08122\") "
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030686    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-certs\") pod \"84119759-f427-4a8d-960e-8b342fca0d38\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") "
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030792    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-envoy\") pod \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") "
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030845    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-token\" (UniqueName: \"kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-token\") pod \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") "
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030930    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-socket\") pod \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") "
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030962    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84119759-f427-4a8d-960e-8b342fca0d38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84119759-f427-4a8d-960e-8b342fca0d38-istiod-ca-cert\") pod \"84119759-f427-4a8d-960e-8b342fca0d38\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") "
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.030992    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-data\") pod \"84119759-f427-4a8d-960e-8b342fca0d38\" (UID: \"84119759-f427-4a8d-960e-8b342fca0d38\") "
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.031030    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-credential-socket\") pod \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\" (UID: \"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38\") "
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.031129    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istiod-ca-cert" pod "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" (UID: "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.031445    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-socket" pod "84119759-f427-4a8d-960e-8b342fca0d38" (UID: "84119759-f427-4a8d-960e-8b342fca0d38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.031796    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-credential-socket" pod "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" (UID: "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.032047    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.032067    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f50a168b-99ca-411d-b32a-19021ca6e029-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.032094    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.032107    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.032120    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.032133    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.032146    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hskn5\" (UniqueName: \"kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-kube-api-access-hskn5\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.032161    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.032176    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f50a168b-99ca-411d-b32a-19021ca6e029-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.032190    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f50a168b-99ca-411d-b32a-19021ca6e029-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.032396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.032203    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f50a168b-99ca-411d-b32a-19021ca6e029-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.033775 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.032454    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-certs" pod "3969f40e-b7c2-4839-85ef-998ab9d08122" (UID: "3969f40e-b7c2-4839-85ef-998ab9d08122"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.033775 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.032863    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-data" pod "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" (UID: "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.033775 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.033076    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-certs" pod "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" (UID: "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.033775 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.033615    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-socket" pod "3969f40e-b7c2-4839-85ef-998ab9d08122" (UID: "3969f40e-b7c2-4839-85ef-998ab9d08122"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.033990 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.033817    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-socket" pod "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" (UID: "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.034264 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.034244    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-credential-socket" pod "84119759-f427-4a8d-960e-8b342fca0d38" (UID: "84119759-f427-4a8d-960e-8b342fca0d38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.034740 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.034721    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-credential-socket" pod "3969f40e-b7c2-4839-85ef-998ab9d08122" (UID: "3969f40e-b7c2-4839-85ef-998ab9d08122"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.037101 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.035774    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-certs" pod "84119759-f427-4a8d-960e-8b342fca0d38" (UID: "84119759-f427-4a8d-960e-8b342fca0d38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.038510 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.038207    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-envoy" pod "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" (UID: "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.038645 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.038590    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-envoy" pod "3969f40e-b7c2-4839-85ef-998ab9d08122" (UID: "3969f40e-b7c2-4839-85ef-998ab9d08122"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.038877 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.038852    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-data" pod "3969f40e-b7c2-4839-85ef-998ab9d08122" (UID: "3969f40e-b7c2-4839-85ef-998ab9d08122"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.039984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.039957    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-kube-api-access-r5pd9" pod "84119759-f427-4a8d-960e-8b342fca0d38" (UID: "84119759-f427-4a8d-960e-8b342fca0d38"). InnerVolumeSpecName "kube-api-access-r5pd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:44.042429 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.042029    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-token" pod "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" (UID: "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:44.042429 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.042165    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-envoy" pod "84119759-f427-4a8d-960e-8b342fca0d38" (UID: "84119759-f427-4a8d-960e-8b342fca0d38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.042429 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.042105    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-podinfo" pod "3969f40e-b7c2-4839-85ef-998ab9d08122" (UID: "3969f40e-b7c2-4839-85ef-998ab9d08122"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:44.042429 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.042325    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-istio-token" pod "84119759-f427-4a8d-960e-8b342fca0d38" (UID: "84119759-f427-4a8d-960e-8b342fca0d38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:44.042775 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.042516    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-podinfo" pod "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" (UID: "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:44.042775 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.042530    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-data" pod "84119759-f427-4a8d-960e-8b342fca0d38" (UID: "84119759-f427-4a8d-960e-8b342fca0d38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:44.042896 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.042858    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3969f40e-b7c2-4839-85ef-998ab9d08122-istiod-ca-cert" pod "3969f40e-b7c2-4839-85ef-998ab9d08122" (UID: "3969f40e-b7c2-4839-85ef-998ab9d08122"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:44.042896 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.042870    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-kube-api-access-sq7z2" pod "3969f40e-b7c2-4839-85ef-998ab9d08122" (UID: "3969f40e-b7c2-4839-85ef-998ab9d08122"). InnerVolumeSpecName "kube-api-access-sq7z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:44.043076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.043053    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-kube-api-access-m94hd" pod "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" (UID: "320a5a00-93dd-4b0d-8bd1-6ba5ed807b38"). InnerVolumeSpecName "kube-api-access-m94hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:44.043225 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.043163    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84119759-f427-4a8d-960e-8b342fca0d38-istiod-ca-cert" pod "84119759-f427-4a8d-960e-8b342fca0d38" (UID: "84119759-f427-4a8d-960e-8b342fca0d38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:44.043975 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.043949    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84119759-f427-4a8d-960e-8b342fca0d38-istio-podinfo" pod "84119759-f427-4a8d-960e-8b342fca0d38" (UID: "84119759-f427-4a8d-960e-8b342fca0d38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:44.044828 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.044806    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-token" pod "3969f40e-b7c2-4839-85ef-998ab9d08122" (UID: "3969f40e-b7c2-4839-85ef-998ab9d08122"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:44.119243 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.118955    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:44.119243 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.119012    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:44.133809 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.133765    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.133995    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84119759-f427-4a8d-960e-8b342fca0d38-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134020    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134033    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134045    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134057    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134069    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134080    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134094    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134107    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134122    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m94hd\" (UniqueName: \"kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-kube-api-access-m94hd\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134135    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3969f40e-b7c2-4839-85ef-998ab9d08122-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134149    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq7z2\" (UniqueName: \"kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-kube-api-access-sq7z2\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134162    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134174    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84119759-f427-4a8d-960e-8b342fca0d38-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134185    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134200    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134211    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3969f40e-b7c2-4839-85ef-998ab9d08122-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.134218 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134223    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.135056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134235    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5pd9\" (UniqueName: \"kubernetes.io/projected/84119759-f427-4a8d-960e-8b342fca0d38-kube-api-access-r5pd9\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.135056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134247    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3969f40e-b7c2-4839-85ef-998ab9d08122-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.135056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134259    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84119759-f427-4a8d-960e-8b342fca0d38-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.135056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134272    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.135056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.134283    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.245400 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.245367    7082 generic.go:356] "Generic (PLEG): container finished" podID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerID="8ce199d2fcbefb880766e3756239e96330e08912f3b858f9ab8d7a9b8d706065" exitCode=0
Oct 29 22:14:44.245910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.245428    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"
Oct 29 22:14:44.245910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.245450    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" event={"ID":"3969f40e-b7c2-4839-85ef-998ab9d08122","Type":"ContainerDied","Data":"8ce199d2fcbefb880766e3756239e96330e08912f3b858f9ab8d7a9b8d706065"}
Oct 29 22:14:44.245910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.245502    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt" event={"ID":"3969f40e-b7c2-4839-85ef-998ab9d08122","Type":"ContainerDied","Data":"88d5403152836dc7510c74273a98bea5b55a66ca2e1d461b50cc73cf67f42b9c"}
Oct 29 22:14:44.245910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.245533    7082 scope.go:122] "RemoveContainer" containerID="8ce199d2fcbefb880766e3756239e96330e08912f3b858f9ab8d7a9b8d706065"
Oct 29 22:14:44.247535 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.247507    7082 generic.go:356] "Generic (PLEG): container finished" podID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerID="e8f11674dc4876ad34473b84353bcee1d2af4b7ad0641a8074cb593b5ec91c45" exitCode=0
Oct 29 22:14:44.247705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.247557    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"
Oct 29 22:14:44.247705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.247595    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" event={"ID":"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38","Type":"ContainerDied","Data":"e8f11674dc4876ad34473b84353bcee1d2af4b7ad0641a8074cb593b5ec91c45"}
Oct 29 22:14:44.247705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.247701    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq" event={"ID":"320a5a00-93dd-4b0d-8bd1-6ba5ed807b38","Type":"ContainerDied","Data":"83df29759509283d88aae061b012f0c33ad41a97a72ad6b3e52784cf2d4cf5f8"}
Oct 29 22:14:44.250148 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.250123    7082 generic.go:356] "Generic (PLEG): container finished" podID="f50a168b-99ca-411d-b32a-19021ca6e029" containerID="99eaa8dd4a77b1b75062415752e29ee672ad20cb2cdbcf1c6bd9bb3bdf2085b5" exitCode=0
Oct 29 22:14:44.250243 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.250216    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" event={"ID":"f50a168b-99ca-411d-b32a-19021ca6e029","Type":"ContainerDied","Data":"99eaa8dd4a77b1b75062415752e29ee672ad20cb2cdbcf1c6bd9bb3bdf2085b5"}
Oct 29 22:14:44.250308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.250255    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q" event={"ID":"f50a168b-99ca-411d-b32a-19021ca6e029","Type":"ContainerDied","Data":"88dfab60c76ff24fc90b752b402d4efa7bac9f4d126b1c42f093730fff478205"}
Oct 29 22:14:44.250382 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.250319    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"
Oct 29 22:14:44.258311 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.258290    7082 generic.go:356] "Generic (PLEG): container finished" podID="84119759-f427-4a8d-960e-8b342fca0d38" containerID="f2ec93756b3a9e6b49a393771834c3f05bb2c5c12e7817d9a7b0a94a48869c8a" exitCode=0
Oct 29 22:14:44.258423 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.258342    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"
Oct 29 22:14:44.258423 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.258377    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" event={"ID":"84119759-f427-4a8d-960e-8b342fca0d38","Type":"ContainerDied","Data":"f2ec93756b3a9e6b49a393771834c3f05bb2c5c12e7817d9a7b0a94a48869c8a"}
Oct 29 22:14:44.258545 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.258426    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd" event={"ID":"84119759-f427-4a8d-960e-8b342fca0d38","Type":"ContainerDied","Data":"6d8aea89d5dc134774fe3387d432b32e4145bacea1f2a038553655688099d161"}
Oct 29 22:14:44.262212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.262187    7082 scope.go:122] "RemoveContainer" containerID="8ce199d2fcbefb880766e3756239e96330e08912f3b858f9ab8d7a9b8d706065"
Oct 29 22:14:44.262337 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.262260    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj" event={"ID":"f69cbc6f-03a9-4469-bfcb-718296f3f8bd","Type":"ContainerStarted","Data":"a9fff270f6751e044807c95c39ae6a0c899808d3f09d8ff8fbc6829ec6f5665e"}
Oct 29 22:14:44.262337 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.262302    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj" event={"ID":"f69cbc6f-03a9-4469-bfcb-718296f3f8bd","Type":"ContainerStarted","Data":"64bc2a0428746fd8e91620d627e1a4bb11e5365237c95515bb3eb037d6fab3c0"}
Oct 29 22:14:44.262900 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:44.262875    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce199d2fcbefb880766e3756239e96330e08912f3b858f9ab8d7a9b8d706065\": container with ID starting with 8ce199d2fcbefb880766e3756239e96330e08912f3b858f9ab8d7a9b8d706065 not found: ID does not exist" containerID="8ce199d2fcbefb880766e3756239e96330e08912f3b858f9ab8d7a9b8d706065"
Oct 29 22:14:44.263031 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.262910    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce199d2fcbefb880766e3756239e96330e08912f3b858f9ab8d7a9b8d706065"} err="failed to get container status \"8ce199d2fcbefb880766e3756239e96330e08912f3b858f9ab8d7a9b8d706065\": rpc error: code = NotFound desc = could not find container \"8ce199d2fcbefb880766e3756239e96330e08912f3b858f9ab8d7a9b8d706065\": container with ID starting with 8ce199d2fcbefb880766e3756239e96330e08912f3b858f9ab8d7a9b8d706065 not found: ID does not exist"
Oct 29 22:14:44.263031 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.262932    7082 scope.go:122] "RemoveContainer" containerID="e8f11674dc4876ad34473b84353bcee1d2af4b7ad0641a8074cb593b5ec91c45"
Oct 29 22:14:44.274721 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.274702    7082 scope.go:122] "RemoveContainer" containerID="e8f11674dc4876ad34473b84353bcee1d2af4b7ad0641a8074cb593b5ec91c45"
Oct 29 22:14:44.275012 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:44.274992    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f11674dc4876ad34473b84353bcee1d2af4b7ad0641a8074cb593b5ec91c45\": container with ID starting with e8f11674dc4876ad34473b84353bcee1d2af4b7ad0641a8074cb593b5ec91c45 not found: ID does not exist" containerID="e8f11674dc4876ad34473b84353bcee1d2af4b7ad0641a8074cb593b5ec91c45"
Oct 29 22:14:44.275102 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.275020    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f11674dc4876ad34473b84353bcee1d2af4b7ad0641a8074cb593b5ec91c45"} err="failed to get container status \"e8f11674dc4876ad34473b84353bcee1d2af4b7ad0641a8074cb593b5ec91c45\": rpc error: code = NotFound desc = could not find container \"e8f11674dc4876ad34473b84353bcee1d2af4b7ad0641a8074cb593b5ec91c45\": container with ID starting with e8f11674dc4876ad34473b84353bcee1d2af4b7ad0641a8074cb593b5ec91c45 not found: ID does not exist"
Oct 29 22:14:44.275102 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.275045    7082 scope.go:122] "RemoveContainer" containerID="99eaa8dd4a77b1b75062415752e29ee672ad20cb2cdbcf1c6bd9bb3bdf2085b5"
Oct 29 22:14:44.285562 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.285543    7082 scope.go:122] "RemoveContainer" containerID="99eaa8dd4a77b1b75062415752e29ee672ad20cb2cdbcf1c6bd9bb3bdf2085b5"
Oct 29 22:14:44.285879 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:44.285857    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99eaa8dd4a77b1b75062415752e29ee672ad20cb2cdbcf1c6bd9bb3bdf2085b5\": container with ID starting with 99eaa8dd4a77b1b75062415752e29ee672ad20cb2cdbcf1c6bd9bb3bdf2085b5 not found: ID does not exist" containerID="99eaa8dd4a77b1b75062415752e29ee672ad20cb2cdbcf1c6bd9bb3bdf2085b5"
Oct 29 22:14:44.285955 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.285886    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99eaa8dd4a77b1b75062415752e29ee672ad20cb2cdbcf1c6bd9bb3bdf2085b5"} err="failed to get container status \"99eaa8dd4a77b1b75062415752e29ee672ad20cb2cdbcf1c6bd9bb3bdf2085b5\": rpc error: code = NotFound desc = could not find container \"99eaa8dd4a77b1b75062415752e29ee672ad20cb2cdbcf1c6bd9bb3bdf2085b5\": container with ID starting with 99eaa8dd4a77b1b75062415752e29ee672ad20cb2cdbcf1c6bd9bb3bdf2085b5 not found: ID does not exist"
Oct 29 22:14:44.285955 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.285907    7082 scope.go:122] "RemoveContainer" containerID="f2ec93756b3a9e6b49a393771834c3f05bb2c5c12e7817d9a7b0a94a48869c8a"
Oct 29 22:14:44.296604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.296588    7082 scope.go:122] "RemoveContainer" containerID="8824abb321ebecc9d637b050ea183c5a3bc42a559dc32b2e6e0254c2320b6290"
Oct 29 22:14:44.310963 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.310941    7082 scope.go:122] "RemoveContainer" containerID="f2ec93756b3a9e6b49a393771834c3f05bb2c5c12e7817d9a7b0a94a48869c8a"
Oct 29 22:14:44.311297 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:44.311269    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ec93756b3a9e6b49a393771834c3f05bb2c5c12e7817d9a7b0a94a48869c8a\": container with ID starting with f2ec93756b3a9e6b49a393771834c3f05bb2c5c12e7817d9a7b0a94a48869c8a not found: ID does not exist" containerID="f2ec93756b3a9e6b49a393771834c3f05bb2c5c12e7817d9a7b0a94a48869c8a"
Oct 29 22:14:44.311391 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.311306    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ec93756b3a9e6b49a393771834c3f05bb2c5c12e7817d9a7b0a94a48869c8a"} err="failed to get container status \"f2ec93756b3a9e6b49a393771834c3f05bb2c5c12e7817d9a7b0a94a48869c8a\": rpc error: code = NotFound desc = could not find container \"f2ec93756b3a9e6b49a393771834c3f05bb2c5c12e7817d9a7b0a94a48869c8a\": container with ID starting with f2ec93756b3a9e6b49a393771834c3f05bb2c5c12e7817d9a7b0a94a48869c8a not found: ID does not exist"
Oct 29 22:14:44.311391 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.311330    7082 scope.go:122] "RemoveContainer" containerID="8824abb321ebecc9d637b050ea183c5a3bc42a559dc32b2e6e0254c2320b6290"
Oct 29 22:14:44.311689 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:44.311662    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8824abb321ebecc9d637b050ea183c5a3bc42a559dc32b2e6e0254c2320b6290\": container with ID starting with 8824abb321ebecc9d637b050ea183c5a3bc42a559dc32b2e6e0254c2320b6290 not found: ID does not exist" containerID="8824abb321ebecc9d637b050ea183c5a3bc42a559dc32b2e6e0254c2320b6290"
Oct 29 22:14:44.311830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.311699    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8824abb321ebecc9d637b050ea183c5a3bc42a559dc32b2e6e0254c2320b6290"} err="failed to get container status \"8824abb321ebecc9d637b050ea183c5a3bc42a559dc32b2e6e0254c2320b6290\": rpc error: code = NotFound desc = could not find container \"8824abb321ebecc9d637b050ea183c5a3bc42a559dc32b2e6e0254c2320b6290\": container with ID starting with 8824abb321ebecc9d637b050ea183c5a3bc42a559dc32b2e6e0254c2320b6290 not found: ID does not exist"
Oct 29 22:14:44.344262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.344101    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"]
Oct 29 22:14:44.348056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.348030    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-wh2tt"]
Oct 29 22:14:44.377489 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.377459    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"]
Oct 29 22:14:44.384114 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.384092    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-bjd5q"]
Oct 29 22:14:44.418073 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.418033    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"]
Oct 29 22:14:44.427395 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.427365    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-2b8gq"]
Oct 29 22:14:44.457333 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.457285    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj" podStartSLOduration=1.457271785 podStartE2EDuration="1.457271785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:44.454726449 +0000 UTC m=+1656.954768746" watchObservedRunningTime="2026-10-29 22:14:44.457271785 +0000 UTC m=+1656.957314084"
Oct 29 22:14:44.497683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.495505    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"]
Oct 29 22:14:44.499744 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.499716    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-vjmbd"]
Oct 29 22:14:44.957622 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.957583    7082 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:44.957748 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:44.957660    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:45.273223 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:45.273126    7082 generic.go:356] "Generic (PLEG): container finished" podID="f69cbc6f-03a9-4469-bfcb-718296f3f8bd" containerID="a9fff270f6751e044807c95c39ae6a0c899808d3f09d8ff8fbc6829ec6f5665e" exitCode=0
Oct 29 22:14:45.273223 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:45.273209    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj" event={"ID":"f69cbc6f-03a9-4469-bfcb-718296f3f8bd","Type":"ContainerDied","Data":"a9fff270f6751e044807c95c39ae6a0c899808d3f09d8ff8fbc6829ec6f5665e"}
Oct 29 22:14:45.633489 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:45.633457    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" path="/var/lib/kubelet/pods/320a5a00-93dd-4b0d-8bd1-6ba5ed807b38/volumes"
Oct 29 22:14:45.634076 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:45.634054    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" path="/var/lib/kubelet/pods/3969f40e-b7c2-4839-85ef-998ab9d08122/volumes"
Oct 29 22:14:45.634566 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:45.634548    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84119759-f427-4a8d-960e-8b342fca0d38" path="/var/lib/kubelet/pods/84119759-f427-4a8d-960e-8b342fca0d38/volumes"
Oct 29 22:14:45.635515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:45.635486    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" path="/var/lib/kubelet/pods/f50a168b-99ca-411d-b32a-19021ca6e029/volumes"
Oct 29 22:14:46.510546 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.510519    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"
Oct 29 22:14:46.572110 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.572065    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-openshift-ca\") pod \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\" (UID: \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\") "
Oct 29 22:14:46.572295 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.572213    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-kube-api-access-ppqpj\" (UniqueName: \"kubernetes.io/projected/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-kube-api-access-ppqpj\") pod \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\" (UID: \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\") "
Oct 29 22:14:46.572490 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.572460    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-openshift-ca" pod "f69cbc6f-03a9-4469-bfcb-718296f3f8bd" (UID: "f69cbc6f-03a9-4469-bfcb-718296f3f8bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:46.572910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.572618    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-xks-ca\") pod \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\" (UID: \"f69cbc6f-03a9-4469-bfcb-718296f3f8bd\") "
Oct 29 22:14:46.573567 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.573546    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.575280 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.575252    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-kube-api-access-ppqpj" pod "f69cbc6f-03a9-4469-bfcb-718296f3f8bd" (UID: "f69cbc6f-03a9-4469-bfcb-718296f3f8bd"). InnerVolumeSpecName "kube-api-access-ppqpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.577267 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.577243    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-xks-ca" pod "f69cbc6f-03a9-4469-bfcb-718296f3f8bd" (UID: "f69cbc6f-03a9-4469-bfcb-718296f3f8bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:46.675595 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.675560    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppqpj\" (UniqueName: \"kubernetes.io/projected/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-kube-api-access-ppqpj\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.675595 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.675598    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69cbc6f-03a9-4469-bfcb-718296f3f8bd-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.813306 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.813276    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 22:14:46.878870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.877419    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-kube-api-access-ckpmt\" (UniqueName: \"kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-kube-api-access-ckpmt\") pod \"1f1ecb02-ee43-4619-9333-a16d536102f8\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") "
Oct 29 22:14:46.878870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.877510    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1ecb02-ee43-4619-9333-a16d536102f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1ecb02-ee43-4619-9333-a16d536102f8-istiod-ca-cert\") pod \"1f1ecb02-ee43-4619-9333-a16d536102f8\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") "
Oct 29 22:14:46.878870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.877618    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-envoy\") pod \"1f1ecb02-ee43-4619-9333-a16d536102f8\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") "
Oct 29 22:14:46.878870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.877710    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-certs\") pod \"1f1ecb02-ee43-4619-9333-a16d536102f8\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") "
Oct 29 22:14:46.878870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.877755    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-socket\") pod \"1f1ecb02-ee43-4619-9333-a16d536102f8\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") "
Oct 29 22:14:46.878870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.877874    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-podinfo\") pod \"1f1ecb02-ee43-4619-9333-a16d536102f8\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") "
Oct 29 22:14:46.878870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.877906    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-credential-socket\") pod \"1f1ecb02-ee43-4619-9333-a16d536102f8\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") "
Oct 29 22:14:46.878870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.877952    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-token\" (UniqueName: \"kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-token\") pod \"1f1ecb02-ee43-4619-9333-a16d536102f8\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") "
Oct 29 22:14:46.878870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.878016    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-data\") pod \"1f1ecb02-ee43-4619-9333-a16d536102f8\" (UID: \"1f1ecb02-ee43-4619-9333-a16d536102f8\") "
Oct 29 22:14:46.879686 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.879655    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-data" pod "1f1ecb02-ee43-4619-9333-a16d536102f8" (UID: "1f1ecb02-ee43-4619-9333-a16d536102f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.880232 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.880163    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-socket" pod "1f1ecb02-ee43-4619-9333-a16d536102f8" (UID: "1f1ecb02-ee43-4619-9333-a16d536102f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.880340 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.880292    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-credential-socket" pod "1f1ecb02-ee43-4619-9333-a16d536102f8" (UID: "1f1ecb02-ee43-4619-9333-a16d536102f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.880586 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.880516    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-certs" pod "1f1ecb02-ee43-4619-9333-a16d536102f8" (UID: "1f1ecb02-ee43-4619-9333-a16d536102f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.880907 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.880874    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1ecb02-ee43-4619-9333-a16d536102f8-istiod-ca-cert" pod "1f1ecb02-ee43-4619-9333-a16d536102f8" (UID: "1f1ecb02-ee43-4619-9333-a16d536102f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:46.882532 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.882505    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-kube-api-access-ckpmt" pod "1f1ecb02-ee43-4619-9333-a16d536102f8" (UID: "1f1ecb02-ee43-4619-9333-a16d536102f8"). InnerVolumeSpecName "kube-api-access-ckpmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.883056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.883035    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-token" pod "1f1ecb02-ee43-4619-9333-a16d536102f8" (UID: "1f1ecb02-ee43-4619-9333-a16d536102f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.883725 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.883698    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-podinfo" pod "1f1ecb02-ee43-4619-9333-a16d536102f8" (UID: "1f1ecb02-ee43-4619-9333-a16d536102f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:46.885417 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.885398    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-envoy" pod "1f1ecb02-ee43-4619-9333-a16d536102f8" (UID: "1f1ecb02-ee43-4619-9333-a16d536102f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.900186 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.900156    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:14:46.903485 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.903460    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:14:46.979978 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.979947    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-podinfo\") pod \"694c4fbc-01cd-4141-8168-2a4c261e0103\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") "
Oct 29 22:14:46.980249 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980043    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-envoy\") pod \"694c4fbc-01cd-4141-8168-2a4c261e0103\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") "
Oct 29 22:14:46.980249 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980179    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-data\") pod \"694c4fbc-01cd-4141-8168-2a4c261e0103\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") "
Oct 29 22:14:46.980249 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980230    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-credential-socket\") pod \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") "
Oct 29 22:14:46.980462 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980276    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/694c4fbc-01cd-4141-8168-2a4c261e0103-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694c4fbc-01cd-4141-8168-2a4c261e0103-istiod-ca-cert\") pod \"694c4fbc-01cd-4141-8168-2a4c261e0103\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") "
Oct 29 22:14:46.980462 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980321    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-socket\") pod \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") "
Oct 29 22:14:46.980462 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980371    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-credential-socket\") pod \"694c4fbc-01cd-4141-8168-2a4c261e0103\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") "
Oct 29 22:14:46.980462 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980418    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-kube-api-access-bwxr4\" (UniqueName: \"kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-kube-api-access-bwxr4\") pod \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") "
Oct 29 22:14:46.980705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980467    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-podinfo\") pod \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") "
Oct 29 22:14:46.980705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980512    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-data\") pod \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") "
Oct 29 22:14:46.980705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980547    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-socket\") pod \"694c4fbc-01cd-4141-8168-2a4c261e0103\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") "
Oct 29 22:14:46.980705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980690    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-credential-socket" pod "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" (UID: "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.980937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980731    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-envoy\") pod \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") "
Oct 29 22:14:46.980937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980789    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-kube-api-access-2whqf\" (UniqueName: \"kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-kube-api-access-2whqf\") pod \"694c4fbc-01cd-4141-8168-2a4c261e0103\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") "
Oct 29 22:14:46.980937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980832    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-token\" (UniqueName: \"kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-token\") pod \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") "
Oct 29 22:14:46.980937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980865    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istiod-ca-cert\") pod \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") "
Oct 29 22:14:46.980937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980891    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-token\" (UniqueName: \"kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-token\") pod \"694c4fbc-01cd-4141-8168-2a4c261e0103\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") "
Oct 29 22:14:46.980937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980909    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-data" pod "694c4fbc-01cd-4141-8168-2a4c261e0103" (UID: "694c4fbc-01cd-4141-8168-2a4c261e0103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.980937 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.980932    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-certs\") pod \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\" (UID: \"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114\") "
Oct 29 22:14:46.981308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981013    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-certs\") pod \"694c4fbc-01cd-4141-8168-2a4c261e0103\" (UID: \"694c4fbc-01cd-4141-8168-2a4c261e0103\") "
Oct 29 22:14:46.981308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981162    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-data" pod "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" (UID: "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.981308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981204    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-credential-socket" pod "694c4fbc-01cd-4141-8168-2a4c261e0103" (UID: "694c4fbc-01cd-4141-8168-2a4c261e0103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.981469 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981363    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694c4fbc-01cd-4141-8168-2a4c261e0103-istiod-ca-cert" pod "694c4fbc-01cd-4141-8168-2a4c261e0103" (UID: "694c4fbc-01cd-4141-8168-2a4c261e0103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:46.981532 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981482    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-socket" pod "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" (UID: "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981846    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-certs" pod "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" (UID: "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981864    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981909    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981903    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istiod-ca-cert" pod "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" (UID: "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981930    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981926    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-certs" pod "694c4fbc-01cd-4141-8168-2a4c261e0103" (UID: "694c4fbc-01cd-4141-8168-2a4c261e0103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981945    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981961    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981974    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981988    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1ecb02-ee43-4619-9333-a16d536102f8-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.981999    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.982015    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.982028    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694c4fbc-01cd-4141-8168-2a4c261e0103-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.982043    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.982056    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckpmt\" (UniqueName: \"kubernetes.io/projected/1f1ecb02-ee43-4619-9333-a16d536102f8-kube-api-access-ckpmt\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.982070    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.982084    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1ecb02-ee43-4619-9333-a16d536102f8-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.982202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.982096    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.984056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.982385    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-socket" pod "694c4fbc-01cd-4141-8168-2a4c261e0103" (UID: "694c4fbc-01cd-4141-8168-2a4c261e0103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.984056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.984020    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-kube-api-access-bwxr4" pod "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" (UID: "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114"). InnerVolumeSpecName "kube-api-access-bwxr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.984357 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.984332    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-envoy" pod "694c4fbc-01cd-4141-8168-2a4c261e0103" (UID: "694c4fbc-01cd-4141-8168-2a4c261e0103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.984716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.984676    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-podinfo" pod "694c4fbc-01cd-4141-8168-2a4c261e0103" (UID: "694c4fbc-01cd-4141-8168-2a4c261e0103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:46.985293 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.985257    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-token" pod "694c4fbc-01cd-4141-8168-2a4c261e0103" (UID: "694c4fbc-01cd-4141-8168-2a4c261e0103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.986072 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.986049    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-podinfo" pod "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" (UID: "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:46.986612 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.986576    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-kube-api-access-2whqf" pod "694c4fbc-01cd-4141-8168-2a4c261e0103" (UID: "694c4fbc-01cd-4141-8168-2a4c261e0103"). InnerVolumeSpecName "kube-api-access-2whqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.987319 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.987272    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-token" pod "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" (UID: "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.987885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:46.987860    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-envoy" pod "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" (UID: "aa1b8cff-4b0b-4b05-b2bc-2ea52741a114"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.083929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.083883    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwxr4\" (UniqueName: \"kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-kube-api-access-bwxr4\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.083929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.083923    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.084151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.083939    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.084151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.083953    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.084151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.083967    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2whqf\" (UniqueName: \"kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-kube-api-access-2whqf\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.084151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.083982    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.084151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.083995    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.084151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.084009    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.084151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.084021    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.084151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.084033    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.084151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.084044    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.084151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.084056    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694c4fbc-01cd-4141-8168-2a4c261e0103-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.309963 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.309931    7082 generic.go:356] "Generic (PLEG): container finished" podID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerID="d1676651cfffbb392920180808eb75a168a8783429a5c2e286d541bbdaf6e009" exitCode=0
Oct 29 22:14:47.310158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.309992    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"
Oct 29 22:14:47.310158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.310013    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" event={"ID":"1f1ecb02-ee43-4619-9333-a16d536102f8","Type":"ContainerDied","Data":"d1676651cfffbb392920180808eb75a168a8783429a5c2e286d541bbdaf6e009"}
Oct 29 22:14:47.310158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.310078    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6" event={"ID":"1f1ecb02-ee43-4619-9333-a16d536102f8","Type":"ContainerDied","Data":"745405b54e2c0f2bde6687ab014320252bfb0d1e70abe7d0e6fec1186cec1063"}
Oct 29 22:14:47.310158 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.310108    7082 scope.go:122] "RemoveContainer" containerID="d1676651cfffbb392920180808eb75a168a8783429a5c2e286d541bbdaf6e009"
Oct 29 22:14:47.314655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.312802    7082 generic.go:356] "Generic (PLEG): container finished" podID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerID="b7f92f156ecb8f1145be58a99a7e5443b9f50a5c0386018c29ce83fd21ccb0ad" exitCode=0
Oct 29 22:14:47.314655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.312914    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" event={"ID":"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114","Type":"ContainerDied","Data":"b7f92f156ecb8f1145be58a99a7e5443b9f50a5c0386018c29ce83fd21ccb0ad"}
Oct 29 22:14:47.314655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.312955    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46" event={"ID":"aa1b8cff-4b0b-4b05-b2bc-2ea52741a114","Type":"ContainerDied","Data":"b30b5eb737191e0cfa1c934e5887f2247fd40d55f733093675ec08fca9a5d7d5"}
Oct 29 22:14:47.314655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.313034    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"
Oct 29 22:14:47.316094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.315148    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj" event={"ID":"f69cbc6f-03a9-4469-bfcb-718296f3f8bd","Type":"ContainerDied","Data":"64bc2a0428746fd8e91620d627e1a4bb11e5365237c95515bb3eb037d6fab3c0"}
Oct 29 22:14:47.316094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.315193    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64bc2a0428746fd8e91620d627e1a4bb11e5365237c95515bb3eb037d6fab3c0"
Oct 29 22:14:47.316094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.315256    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"
Oct 29 22:14:47.321497 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.321468    7082 generic.go:356] "Generic (PLEG): container finished" podID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerID="75abdf7b32b7a08079db296a280754d8aca7410046713628c21a0189290f5d38" exitCode=0
Oct 29 22:14:47.321938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.321751    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" event={"ID":"694c4fbc-01cd-4141-8168-2a4c261e0103","Type":"ContainerDied","Data":"75abdf7b32b7a08079db296a280754d8aca7410046713628c21a0189290f5d38"}
Oct 29 22:14:47.321938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.321819    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h" event={"ID":"694c4fbc-01cd-4141-8168-2a4c261e0103","Type":"ContainerDied","Data":"2bf2e9a41218fa26d028a650c6baf6f69d7e31686c224136d046b1b7653865a0"}
Oct 29 22:14:47.321938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.321911    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"
Oct 29 22:14:47.327244 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.327021    7082 scope.go:122] "RemoveContainer" containerID="d1676651cfffbb392920180808eb75a168a8783429a5c2e286d541bbdaf6e009"
Oct 29 22:14:47.328003 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:47.327607    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1676651cfffbb392920180808eb75a168a8783429a5c2e286d541bbdaf6e009\": container with ID starting with d1676651cfffbb392920180808eb75a168a8783429a5c2e286d541bbdaf6e009 not found: ID does not exist" containerID="d1676651cfffbb392920180808eb75a168a8783429a5c2e286d541bbdaf6e009"
Oct 29 22:14:47.328003 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.327660    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1676651cfffbb392920180808eb75a168a8783429a5c2e286d541bbdaf6e009"} err="failed to get container status \"d1676651cfffbb392920180808eb75a168a8783429a5c2e286d541bbdaf6e009\": rpc error: code = NotFound desc = could not find container \"d1676651cfffbb392920180808eb75a168a8783429a5c2e286d541bbdaf6e009\": container with ID starting with d1676651cfffbb392920180808eb75a168a8783429a5c2e286d541bbdaf6e009 not found: ID does not exist"
Oct 29 22:14:47.328003 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.327683    7082 scope.go:122] "RemoveContainer" containerID="b7f92f156ecb8f1145be58a99a7e5443b9f50a5c0386018c29ce83fd21ccb0ad"
Oct 29 22:14:47.341386 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.341369    7082 scope.go:122] "RemoveContainer" containerID="b7f92f156ecb8f1145be58a99a7e5443b9f50a5c0386018c29ce83fd21ccb0ad"
Oct 29 22:14:47.341693 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:47.341665    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f92f156ecb8f1145be58a99a7e5443b9f50a5c0386018c29ce83fd21ccb0ad\": container with ID starting with b7f92f156ecb8f1145be58a99a7e5443b9f50a5c0386018c29ce83fd21ccb0ad not found: ID does not exist" containerID="b7f92f156ecb8f1145be58a99a7e5443b9f50a5c0386018c29ce83fd21ccb0ad"
Oct 29 22:14:47.341773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.341702    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f92f156ecb8f1145be58a99a7e5443b9f50a5c0386018c29ce83fd21ccb0ad"} err="failed to get container status \"b7f92f156ecb8f1145be58a99a7e5443b9f50a5c0386018c29ce83fd21ccb0ad\": rpc error: code = NotFound desc = could not find container \"b7f92f156ecb8f1145be58a99a7e5443b9f50a5c0386018c29ce83fd21ccb0ad\": container with ID starting with b7f92f156ecb8f1145be58a99a7e5443b9f50a5c0386018c29ce83fd21ccb0ad not found: ID does not exist"
Oct 29 22:14:47.341773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.341723    7082 scope.go:122] "RemoveContainer" containerID="75abdf7b32b7a08079db296a280754d8aca7410046713628c21a0189290f5d38"
Oct 29 22:14:47.352439 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.352407    7082 scope.go:122] "RemoveContainer" containerID="75abdf7b32b7a08079db296a280754d8aca7410046713628c21a0189290f5d38"
Oct 29 22:14:47.352771 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:47.352750    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75abdf7b32b7a08079db296a280754d8aca7410046713628c21a0189290f5d38\": container with ID starting with 75abdf7b32b7a08079db296a280754d8aca7410046713628c21a0189290f5d38 not found: ID does not exist" containerID="75abdf7b32b7a08079db296a280754d8aca7410046713628c21a0189290f5d38"
Oct 29 22:14:47.352860 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.352776    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75abdf7b32b7a08079db296a280754d8aca7410046713628c21a0189290f5d38"} err="failed to get container status \"75abdf7b32b7a08079db296a280754d8aca7410046713628c21a0189290f5d38\": rpc error: code = NotFound desc = could not find container \"75abdf7b32b7a08079db296a280754d8aca7410046713628c21a0189290f5d38\": container with ID starting with 75abdf7b32b7a08079db296a280754d8aca7410046713628c21a0189290f5d38 not found: ID does not exist"
Oct 29 22:14:47.382040 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.382017    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"]
Oct 29 22:14:47.397900 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.396670    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-j6zg6"]
Oct 29 22:14:47.439657 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.439575    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"]
Oct 29 22:14:47.445336 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.445314    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-xd57h"]
Oct 29 22:14:47.477908 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.477884    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"]
Oct 29 22:14:47.482702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.482675    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-tgj46"]
Oct 29 22:14:47.634652 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.634163    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" path="/var/lib/kubelet/pods/1f1ecb02-ee43-4619-9333-a16d536102f8/volumes"
Oct 29 22:14:47.635117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.634696    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" path="/var/lib/kubelet/pods/694c4fbc-01cd-4141-8168-2a4c261e0103/volumes"
Oct 29 22:14:47.635592 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.635314    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" path="/var/lib/kubelet/pods/aa1b8cff-4b0b-4b05-b2bc-2ea52741a114/volumes"
Oct 29 22:14:47.757241 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.757210    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:14:47.812649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.811775    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-credential-socket\") pod \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") "
Oct 29 22:14:47.812649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.811854    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-podinfo\") pod \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") "
Oct 29 22:14:47.812649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.812104    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-credential-socket" pod "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" (UID: "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.812649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.812293    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-token\" (UniqueName: \"kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-token\") pod \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") "
Oct 29 22:14:47.812649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.812351    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-socket\") pod \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") "
Oct 29 22:14:47.812649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.812390    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-kube-api-access-4kk29\" (UniqueName: \"kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-kube-api-access-4kk29\") pod \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") "
Oct 29 22:14:47.812649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.812425    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-certs\") pod \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") "
Oct 29 22:14:47.812649 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.812514    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-data\") pod \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") "
Oct 29 22:14:47.813223 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.812669    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-socket" pod "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" (UID: "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.813223 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.812693    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istiod-ca-cert\") pod \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") "
Oct 29 22:14:47.813223 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.812729    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-envoy\") pod \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\" (UID: \"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc\") "
Oct 29 22:14:47.813223 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.812856    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-certs" pod "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" (UID: "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.813431 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.813320    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.813431 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.813342    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.813431 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.813356    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.820677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.814207    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-data" pod "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" (UID: "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.820677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.814356    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istiod-ca-cert" pod "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" (UID: "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:47.820677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.819827    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-podinfo" pod "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" (UID: "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:47.820677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.819980    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-kube-api-access-4kk29" pod "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" (UID: "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc"). InnerVolumeSpecName "kube-api-access-4kk29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:47.820677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.820230    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-envoy" pod "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" (UID: "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.820677 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.820605    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-token" pod "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" (UID: "bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:47.909197 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.909104    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"]
Oct 29 22:14:47.914920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.914889    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.914920 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.914922    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.915092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.914936    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kk29\" (UniqueName: \"kubernetes.io/projected/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-kube-api-access-4kk29\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.915092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.914952    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.915092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.914966    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.915092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:47.914978    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:48.114115 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.114090    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:14:48.221324 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.221237    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-credential-socket\") pod \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") "
Oct 29 22:14:48.221324 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.221288    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-envoy\") pod \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") "
Oct 29 22:14:48.221536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.221364    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-data\") pod \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") "
Oct 29 22:14:48.221536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.221494    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-token\" (UniqueName: \"kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-token\") pod \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") "
Oct 29 22:14:48.221664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.221547    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-credential-socket" pod "027d1e1a-2888-45b7-b47e-2b62ca191f7a" (UID: "027d1e1a-2888-45b7-b47e-2b62ca191f7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:48.221664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.221562    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-certs\") pod \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") "
Oct 29 22:14:48.221664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.221595    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-kube-api-access-gs47w\" (UniqueName: \"kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-kube-api-access-gs47w\") pod \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") "
Oct 29 22:14:48.221840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.221717    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istiod-ca-cert\") pod \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") "
Oct 29 22:14:48.221840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.221798    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-data" pod "027d1e1a-2888-45b7-b47e-2b62ca191f7a" (UID: "027d1e1a-2888-45b7-b47e-2b62ca191f7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:48.221840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.221804    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-socket\") pod \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") "
Oct 29 22:14:48.221998 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.221848    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-certs" pod "027d1e1a-2888-45b7-b47e-2b62ca191f7a" (UID: "027d1e1a-2888-45b7-b47e-2b62ca191f7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:48.221998 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.221860    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-podinfo\") pod \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\" (UID: \"027d1e1a-2888-45b7-b47e-2b62ca191f7a\") "
Oct 29 22:14:48.222118 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.222021    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-socket" pod "027d1e1a-2888-45b7-b47e-2b62ca191f7a" (UID: "027d1e1a-2888-45b7-b47e-2b62ca191f7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:48.222206 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.222180    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istiod-ca-cert" pod "027d1e1a-2888-45b7-b47e-2b62ca191f7a" (UID: "027d1e1a-2888-45b7-b47e-2b62ca191f7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:48.222622 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.222595    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:48.222746 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.222663    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:48.222746 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.222679    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:48.222746 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.222691    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:48.222746 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.222702    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:48.224144 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.224122    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-envoy" pod "027d1e1a-2888-45b7-b47e-2b62ca191f7a" (UID: "027d1e1a-2888-45b7-b47e-2b62ca191f7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:48.224671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.224618    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-kube-api-access-gs47w" pod "027d1e1a-2888-45b7-b47e-2b62ca191f7a" (UID: "027d1e1a-2888-45b7-b47e-2b62ca191f7a"). InnerVolumeSpecName "kube-api-access-gs47w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:48.224671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.224618    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-podinfo" pod "027d1e1a-2888-45b7-b47e-2b62ca191f7a" (UID: "027d1e1a-2888-45b7-b47e-2b62ca191f7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:48.224799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.224737    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-token" pod "027d1e1a-2888-45b7-b47e-2b62ca191f7a" (UID: "027d1e1a-2888-45b7-b47e-2b62ca191f7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:48.324431 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.324398    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:48.324431 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.324430    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:48.324638 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.324444    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs47w\" (UniqueName: \"kubernetes.io/projected/027d1e1a-2888-45b7-b47e-2b62ca191f7a-kube-api-access-gs47w\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:48.324638 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.324458    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/027d1e1a-2888-45b7-b47e-2b62ca191f7a-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:48.333109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.333076    7082 generic.go:356] "Generic (PLEG): container finished" podID="027d1e1a-2888-45b7-b47e-2b62ca191f7a" containerID="32814cff4a4183d3ce8a8114c365e76cdefaeb5dbad2d2d7190c63cbc07fa82a" exitCode=0
Oct 29 22:14:48.333268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.333130    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"
Oct 29 22:14:48.333268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.333160    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz" event={"ID":"027d1e1a-2888-45b7-b47e-2b62ca191f7a","Type":"ContainerDied","Data":"32814cff4a4183d3ce8a8114c365e76cdefaeb5dbad2d2d7190c63cbc07fa82a"}
Oct 29 22:14:48.333268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.333215    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz" event={"ID":"027d1e1a-2888-45b7-b47e-2b62ca191f7a","Type":"ContainerDied","Data":"ab6455286795e0e25599c7b0704c02d5ca5eaeac8906fe31e84547ecc3f224e2"}
Oct 29 22:14:48.333268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.333240    7082 scope.go:122] "RemoveContainer" containerID="32814cff4a4183d3ce8a8114c365e76cdefaeb5dbad2d2d7190c63cbc07fa82a"
Oct 29 22:14:48.335139 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.335114    7082 generic.go:356] "Generic (PLEG): container finished" podID="bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" containerID="6ff3769da58d33e970c6e2f32744eb238e97eebefee07dfe880d9b0f660a61f3" exitCode=0
Oct 29 22:14:48.335266 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.335160    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"
Oct 29 22:14:48.335266 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.335196    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb" event={"ID":"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc","Type":"ContainerDied","Data":"6ff3769da58d33e970c6e2f32744eb238e97eebefee07dfe880d9b0f660a61f3"}
Oct 29 22:14:48.335266 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.335243    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb" event={"ID":"bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc","Type":"ContainerDied","Data":"38f662168623aa603f410ebd3eb9f89c7bf598a6f18e6df62fa4f161fb3b9cb6"}
Oct 29 22:14:48.347814 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.347797    7082 scope.go:122] "RemoveContainer" containerID="32814cff4a4183d3ce8a8114c365e76cdefaeb5dbad2d2d7190c63cbc07fa82a"
Oct 29 22:14:48.348072 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:48.348054    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32814cff4a4183d3ce8a8114c365e76cdefaeb5dbad2d2d7190c63cbc07fa82a\": container with ID starting with 32814cff4a4183d3ce8a8114c365e76cdefaeb5dbad2d2d7190c63cbc07fa82a not found: ID does not exist" containerID="32814cff4a4183d3ce8a8114c365e76cdefaeb5dbad2d2d7190c63cbc07fa82a"
Oct 29 22:14:48.348133 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.348091    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32814cff4a4183d3ce8a8114c365e76cdefaeb5dbad2d2d7190c63cbc07fa82a"} err="failed to get container status \"32814cff4a4183d3ce8a8114c365e76cdefaeb5dbad2d2d7190c63cbc07fa82a\": rpc error: code = NotFound desc = could not find container \"32814cff4a4183d3ce8a8114c365e76cdefaeb5dbad2d2d7190c63cbc07fa82a\": container with ID starting with 32814cff4a4183d3ce8a8114c365e76cdefaeb5dbad2d2d7190c63cbc07fa82a not found: ID does not exist"
Oct 29 22:14:48.348133 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.348108    7082 scope.go:122] "RemoveContainer" containerID="6ff3769da58d33e970c6e2f32744eb238e97eebefee07dfe880d9b0f660a61f3"
Oct 29 22:14:48.358576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.358561    7082 scope.go:122] "RemoveContainer" containerID="6ff3769da58d33e970c6e2f32744eb238e97eebefee07dfe880d9b0f660a61f3"
Oct 29 22:14:48.358842 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:48.358824    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff3769da58d33e970c6e2f32744eb238e97eebefee07dfe880d9b0f660a61f3\": container with ID starting with 6ff3769da58d33e970c6e2f32744eb238e97eebefee07dfe880d9b0f660a61f3 not found: ID does not exist" containerID="6ff3769da58d33e970c6e2f32744eb238e97eebefee07dfe880d9b0f660a61f3"
Oct 29 22:14:48.358887 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.358849    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff3769da58d33e970c6e2f32744eb238e97eebefee07dfe880d9b0f660a61f3"} err="failed to get container status \"6ff3769da58d33e970c6e2f32744eb238e97eebefee07dfe880d9b0f660a61f3\": rpc error: code = NotFound desc = could not find container \"6ff3769da58d33e970c6e2f32744eb238e97eebefee07dfe880d9b0f660a61f3\": container with ID starting with 6ff3769da58d33e970c6e2f32744eb238e97eebefee07dfe880d9b0f660a61f3 not found: ID does not exist"
Oct 29 22:14:48.384511 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.384485    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"]
Oct 29 22:14:48.388003 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.387981    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-wt6hz"]
Oct 29 22:14:48.420845 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.420823    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"]
Oct 29 22:14:48.426149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:48.426125    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-zfmhb"]
Oct 29 22:14:49.633923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:49.633891    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027d1e1a-2888-45b7-b47e-2b62ca191f7a" path="/var/lib/kubelet/pods/027d1e1a-2888-45b7-b47e-2b62ca191f7a/volumes"
Oct 29 22:14:49.634565 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:49.634544    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" path="/var/lib/kubelet/pods/bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc/volumes"
Oct 29 22:14:50.429282 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:50.429248    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"]
Oct 29 22:14:50.434642 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:50.434604    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f363a3-fhcg8"]
Oct 29 22:14:51.631441 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:51.631403    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ec21198-9a7b-44b1-af11-aa6a89c71c7c" path="/var/lib/kubelet/pods/3ec21198-9a7b-44b1-af11-aa6a89c71c7c/volumes"
Oct 29 22:14:55.632324 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.632291    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"]
Oct 29 22:14:55.632922 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.632905    7082 cpu_manager.go:414] "containerMap: removing container" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy"
Oct 29 22:14:55.633002 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.632925    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy"
Oct 29 22:14:55.633002 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.632945    7082 cpu_manager.go:414] "containerMap: removing container" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy"
Oct 29 22:14:55.633002 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.632950    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy"
Oct 29 22:14:55.633002 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.632961    7082 cpu_manager.go:414] "containerMap: removing container" podUID="f69cbc6f-03a9-4469-bfcb-718296f3f8bd" containerName="revoke-keys"
Oct 29 22:14:55.633002 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.632966    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69cbc6f-03a9-4469-bfcb-718296f3f8bd" containerName="revoke-keys"
Oct 29 22:14:55.633002 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.632973    7082 cpu_manager.go:414] "containerMap: removing container" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy"
Oct 29 22:14:55.633002 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.632978    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy"
Oct 29 22:14:55.633002 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.632988    7082 cpu_manager.go:414] "containerMap: removing container" podUID="027d1e1a-2888-45b7-b47e-2b62ca191f7a" containerName="istio-proxy"
Oct 29 22:14:55.633002 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.632993    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027d1e1a-2888-45b7-b47e-2b62ca191f7a" containerName="istio-proxy"
Oct 29 22:14:55.633002 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633005    7082 cpu_manager.go:414] "containerMap: removing container" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633013    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633029    7082 cpu_manager.go:414] "containerMap: removing container" podUID="bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633033    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633053    7082 cpu_manager.go:414] "containerMap: removing container" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633057    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633069    7082 cpu_manager.go:414] "containerMap: removing container" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633073    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633085    7082 cpu_manager.go:414] "containerMap: removing container" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633092    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633181    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="694c4fbc-01cd-4141-8168-2a4c261e0103" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633192    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa1b8cff-4b0b-4b05-b2bc-2ea52741a114" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633198    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="3969f40e-b7c2-4839-85ef-998ab9d08122" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633205    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="f50a168b-99ca-411d-b32a-19021ca6e029" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633210    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633215    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69cbc6f-03a9-4469-bfcb-718296f3f8bd" containerName="revoke-keys"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633221    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="320a5a00-93dd-4b0d-8bd1-6ba5ed807b38" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633227    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="027d1e1a-2888-45b7-b47e-2b62ca191f7a" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633232    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1ecb02-ee43-4619-9333-a16d536102f8" containerName="istio-proxy"
Oct 29 22:14:55.633318 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.633240    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf0e7932-be97-4cbb-b6dd-c514b1ba6dcc" containerName="istio-proxy"
Oct 29 22:14:55.639425 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.639406    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"
Oct 29 22:14:55.657799 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.657775    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"]
Oct 29 22:14:55.659251 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.659228    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:14:55.659251 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.659235    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:14:55.668156 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.665571    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:14:55.668363 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.668348    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:14:55.683020 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.682972    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn2f2\" (UniqueName: \"kubernetes.io/projected/4646467d-1071-4a13-908c-42cadafb711c-kube-api-access-dn2f2\") pod \"maas-api-revoke-sub-75d26cc0-prwlq\" (UID: \"4646467d-1071-4a13-908c-42cadafb711c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"
Oct 29 22:14:55.683217 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.683155    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4646467d-1071-4a13-908c-42cadafb711c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d26cc0-prwlq\" (UID: \"4646467d-1071-4a13-908c-42cadafb711c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"
Oct 29 22:14:55.683420 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.683365    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4646467d-1071-4a13-908c-42cadafb711c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d26cc0-prwlq\" (UID: \"4646467d-1071-4a13-908c-42cadafb711c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"
Oct 29 22:14:55.784242 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.784209    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn2f2\" (UniqueName: \"kubernetes.io/projected/4646467d-1071-4a13-908c-42cadafb711c-kube-api-access-dn2f2\") pod \"maas-api-revoke-sub-75d26cc0-prwlq\" (UID: \"4646467d-1071-4a13-908c-42cadafb711c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"
Oct 29 22:14:55.784410 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.784311    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4646467d-1071-4a13-908c-42cadafb711c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d26cc0-prwlq\" (UID: \"4646467d-1071-4a13-908c-42cadafb711c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"
Oct 29 22:14:55.784410 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.784405    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4646467d-1071-4a13-908c-42cadafb711c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d26cc0-prwlq\" (UID: \"4646467d-1071-4a13-908c-42cadafb711c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"
Oct 29 22:14:55.785072 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.785045    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4646467d-1071-4a13-908c-42cadafb711c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75d26cc0-prwlq\" (UID: \"4646467d-1071-4a13-908c-42cadafb711c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"
Oct 29 22:14:55.786926 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.786901    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4646467d-1071-4a13-908c-42cadafb711c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75d26cc0-prwlq\" (UID: \"4646467d-1071-4a13-908c-42cadafb711c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"
Oct 29 22:14:55.807075 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.807049    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn2f2\" (UniqueName: \"kubernetes.io/projected/4646467d-1071-4a13-908c-42cadafb711c-kube-api-access-dn2f2\") pod \"maas-api-revoke-sub-75d26cc0-prwlq\" (UID: \"4646467d-1071-4a13-908c-42cadafb711c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"
Oct 29 22:14:55.956655 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:55.956526    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"
Oct 29 22:14:56.268456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:56.268430    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"]
Oct 29 22:14:56.270006 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:14:56.269973    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4646467d_1071_4a13_908c_42cadafb711c.slice/crio-e9636f46998203b565fa0c051bcadd68d0c8e2e25efffe0282eca9b28d4952a5 WatchSource:0}: Error finding container e9636f46998203b565fa0c051bcadd68d0c8e2e25efffe0282eca9b28d4952a5: Status 404 returned error can't find the container with id e9636f46998203b565fa0c051bcadd68d0c8e2e25efffe0282eca9b28d4952a5
Oct 29 22:14:56.421321 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:56.421288    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq" event={"ID":"4646467d-1071-4a13-908c-42cadafb711c","Type":"ContainerStarted","Data":"e9636f46998203b565fa0c051bcadd68d0c8e2e25efffe0282eca9b28d4952a5"}
Oct 29 22:14:57.431211 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:57.431178    7082 generic.go:356] "Generic (PLEG): container finished" podID="4646467d-1071-4a13-908c-42cadafb711c" containerID="d62787ebe76fe0f565b6b3ff363b36ef08abf558c9a452ebfe61658d54ff571b" exitCode=0
Oct 29 22:14:57.431697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:57.431259    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq" event={"ID":"4646467d-1071-4a13-908c-42cadafb711c","Type":"ContainerDied","Data":"d62787ebe76fe0f565b6b3ff363b36ef08abf558c9a452ebfe61658d54ff571b"}
Oct 29 22:14:58.415582 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:14:58.415541    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:14:58.596015 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:58.595988    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"
Oct 29 22:14:58.623691 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:58.623660    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4646467d-1071-4a13-908c-42cadafb711c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4646467d-1071-4a13-908c-42cadafb711c-maas-api-openshift-ca\") pod \"4646467d-1071-4a13-908c-42cadafb711c\" (UID: \"4646467d-1071-4a13-908c-42cadafb711c\") "
Oct 29 22:14:58.623830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:58.623744    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4646467d-1071-4a13-908c-42cadafb711c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4646467d-1071-4a13-908c-42cadafb711c-maas-api-xks-ca\") pod \"4646467d-1071-4a13-908c-42cadafb711c\" (UID: \"4646467d-1071-4a13-908c-42cadafb711c\") "
Oct 29 22:14:58.623830 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:58.623797    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4646467d-1071-4a13-908c-42cadafb711c-kube-api-access-dn2f2\" (UniqueName: \"kubernetes.io/projected/4646467d-1071-4a13-908c-42cadafb711c-kube-api-access-dn2f2\") pod \"4646467d-1071-4a13-908c-42cadafb711c\" (UID: \"4646467d-1071-4a13-908c-42cadafb711c\") "
Oct 29 22:14:58.623998 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:58.623972    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4646467d-1071-4a13-908c-42cadafb711c-maas-api-openshift-ca" pod "4646467d-1071-4a13-908c-42cadafb711c" (UID: "4646467d-1071-4a13-908c-42cadafb711c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:58.624300 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:58.624267    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4646467d-1071-4a13-908c-42cadafb711c-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:58.626515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:58.626490    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4646467d-1071-4a13-908c-42cadafb711c-maas-api-xks-ca" pod "4646467d-1071-4a13-908c-42cadafb711c" (UID: "4646467d-1071-4a13-908c-42cadafb711c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:58.626515 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:58.626507    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4646467d-1071-4a13-908c-42cadafb711c-kube-api-access-dn2f2" pod "4646467d-1071-4a13-908c-42cadafb711c" (UID: "4646467d-1071-4a13-908c-42cadafb711c"). InnerVolumeSpecName "kube-api-access-dn2f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:58.726079 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:58.726002    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4646467d-1071-4a13-908c-42cadafb711c-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:58.726079 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:58.726041    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn2f2\" (UniqueName: \"kubernetes.io/projected/4646467d-1071-4a13-908c-42cadafb711c-kube-api-access-dn2f2\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:14:59.449614 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:59.449585    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq"
Oct 29 22:14:59.449614 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:59.449613    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75d26cc0-prwlq" event={"ID":"4646467d-1071-4a13-908c-42cadafb711c","Type":"ContainerDied","Data":"e9636f46998203b565fa0c051bcadd68d0c8e2e25efffe0282eca9b28d4952a5"}
Oct 29 22:14:59.449866 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:14:59.449678    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9636f46998203b565fa0c051bcadd68d0c8e2e25efffe0282eca9b28d4952a5"
Oct 29 22:15:00.144978 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.144939    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"]
Oct 29 22:15:00.145525 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.145510    7082 cpu_manager.go:414] "containerMap: removing container" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy"
Oct 29 22:15:00.145525 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.145524    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy"
Oct 29 22:15:00.145704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.145564    7082 cpu_manager.go:414] "containerMap: removing container" podUID="4646467d-1071-4a13-908c-42cadafb711c" containerName="revoke-keys"
Oct 29 22:15:00.145704 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.145570    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4646467d-1071-4a13-908c-42cadafb711c" containerName="revoke-keys"
Oct 29 22:15:00.145844 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.145719    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="4646467d-1071-4a13-908c-42cadafb711c" containerName="revoke-keys"
Oct 29 22:15:00.145844 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.145742    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="84119759-f427-4a8d-960e-8b342fca0d38" containerName="istio-proxy"
Oct 29 22:15:00.149639 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.149599    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"
Oct 29 22:15:00.156429 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.156391    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"]
Oct 29 22:15:00.202651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.178596    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"
Oct 29 22:15:00.207534 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.193239    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:15:00.207736 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.193314    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:15:00.207827 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.193385    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:15:00.266661 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.265192    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"]
Oct 29 22:15:00.300773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.300564    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"]
Oct 29 22:15:00.300773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.300664    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:15:00.304208 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.304088    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf70914c-77a1-427d-8c85-903954bde9dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252\" (UID: \"bf70914c-77a1-427d-8c85-903954bde9dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"
Oct 29 22:15:00.304402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.304213    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7380f6d-cac7-48e0-b312-41612a211252-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v\" (UID: \"f7380f6d-cac7-48e0-b312-41612a211252\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"
Oct 29 22:15:00.304402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.304280    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9x5k\" (UniqueName: \"kubernetes.io/projected/f7380f6d-cac7-48e0-b312-41612a211252-kube-api-access-t9x5k\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v\" (UID: \"f7380f6d-cac7-48e0-b312-41612a211252\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"
Oct 29 22:15:00.304402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.304382    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvb8s\" (UniqueName: \"kubernetes.io/projected/bf70914c-77a1-427d-8c85-903954bde9dd-kube-api-access-lvb8s\") pod \"maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252\" (UID: \"bf70914c-77a1-427d-8c85-903954bde9dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"
Oct 29 22:15:00.405562 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.405469    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvb8s\" (UniqueName: \"kubernetes.io/projected/bf70914c-77a1-427d-8c85-903954bde9dd-kube-api-access-lvb8s\") pod \"maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252\" (UID: \"bf70914c-77a1-427d-8c85-903954bde9dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"
Oct 29 22:15:00.405783 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.405686    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf70914c-77a1-427d-8c85-903954bde9dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252\" (UID: \"bf70914c-77a1-427d-8c85-903954bde9dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"
Oct 29 22:15:00.405783 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.405770    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7380f6d-cac7-48e0-b312-41612a211252-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v\" (UID: \"f7380f6d-cac7-48e0-b312-41612a211252\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"
Oct 29 22:15:00.405917 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.405825    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9x5k\" (UniqueName: \"kubernetes.io/projected/f7380f6d-cac7-48e0-b312-41612a211252-kube-api-access-t9x5k\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v\" (UID: \"f7380f6d-cac7-48e0-b312-41612a211252\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"
Oct 29 22:15:00.406554 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.406518    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7380f6d-cac7-48e0-b312-41612a211252-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v\" (UID: \"f7380f6d-cac7-48e0-b312-41612a211252\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"
Oct 29 22:15:00.406753 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.406732    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf70914c-77a1-427d-8c85-903954bde9dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252\" (UID: \"bf70914c-77a1-427d-8c85-903954bde9dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"
Oct 29 22:15:00.428836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.428811    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9x5k\" (UniqueName: \"kubernetes.io/projected/f7380f6d-cac7-48e0-b312-41612a211252-kube-api-access-t9x5k\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v\" (UID: \"f7380f6d-cac7-48e0-b312-41612a211252\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"
Oct 29 22:15:00.429292 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.429269    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvb8s\" (UniqueName: \"kubernetes.io/projected/bf70914c-77a1-427d-8c85-903954bde9dd-kube-api-access-lvb8s\") pod \"maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252\" (UID: \"bf70914c-77a1-427d-8c85-903954bde9dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"
Oct 29 22:15:00.512792 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.512761    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"
Oct 29 22:15:00.635367 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.619955    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"
Oct 29 22:15:00.845368 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:00.845341    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"]
Oct 29 22:15:00.849293 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:15:00.849069    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7380f6d_cac7_48e0_b312_41612a211252.slice/crio-7d896549e0a9ca41a1949a3d8aa8229b8c379a354eef4a3954366b15e10ad984 WatchSource:0}: Error finding container 7d896549e0a9ca41a1949a3d8aa8229b8c379a354eef4a3954366b15e10ad984: Status 404 returned error can't find the container with id 7d896549e0a9ca41a1949a3d8aa8229b8c379a354eef4a3954366b15e10ad984
Oct 29 22:15:01.046123 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:01.046096    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"]
Oct 29 22:15:01.047333 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:15:01.047298    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf70914c_77a1_427d_8c85_903954bde9dd.slice/crio-2f23d0cb2ab91991f241db92a4ce549570638fed6f23d8a445684e8d32401cd4 WatchSource:0}: Error finding container 2f23d0cb2ab91991f241db92a4ce549570638fed6f23d8a445684e8d32401cd4: Status 404 returned error can't find the container with id 2f23d0cb2ab91991f241db92a4ce549570638fed6f23d8a445684e8d32401cd4
Oct 29 22:15:01.539980 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:01.535125    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252" event={"ID":"bf70914c-77a1-427d-8c85-903954bde9dd","Type":"ContainerStarted","Data":"eadbbd08eb04c8129c4447170b111d1088cd459e0ed0f93ae6dc581d427cc9f8"}
Oct 29 22:15:01.539980 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:01.535197    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252" event={"ID":"bf70914c-77a1-427d-8c85-903954bde9dd","Type":"ContainerStarted","Data":"2f23d0cb2ab91991f241db92a4ce549570638fed6f23d8a445684e8d32401cd4"}
Oct 29 22:15:01.618357 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:01.616188    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v" event={"ID":"f7380f6d-cac7-48e0-b312-41612a211252","Type":"ContainerStarted","Data":"673e46677836cde5d3a5b812d8b6fedc54455c4cb692fa3ba8c594e5ca02d7c8"}
Oct 29 22:15:01.618357 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:01.616299    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v" event={"ID":"f7380f6d-cac7-48e0-b312-41612a211252","Type":"ContainerStarted","Data":"7d896549e0a9ca41a1949a3d8aa8229b8c379a354eef4a3954366b15e10ad984"}
Oct 29 22:15:01.623006 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:01.622947    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252" podStartSLOduration=1.6229277789999998 podStartE2EDuration="1.622927779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:01.600241925 +0000 UTC m=+1674.100284228" watchObservedRunningTime="2026-10-29 22:15:01.622927779 +0000 UTC m=+1674.122970084"
Oct 29 22:15:01.664897 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:01.664832    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v" podStartSLOduration=1.6648138019999998 podStartE2EDuration="1.664813802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:01.66171967 +0000 UTC m=+1674.161761970" watchObservedRunningTime="2026-10-29 22:15:01.664813802 +0000 UTC m=+1674.164856101"
Oct 29 22:15:02.651680 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:02.651563    7082 generic.go:356] "Generic (PLEG): container finished" podID="bf70914c-77a1-427d-8c85-903954bde9dd" containerID="eadbbd08eb04c8129c4447170b111d1088cd459e0ed0f93ae6dc581d427cc9f8" exitCode=60
Oct 29 22:15:02.652223 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:02.651761    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252" event={"ID":"bf70914c-77a1-427d-8c85-903954bde9dd","Type":"ContainerDied","Data":"eadbbd08eb04c8129c4447170b111d1088cd459e0ed0f93ae6dc581d427cc9f8"}
Oct 29 22:15:02.652223 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:02.652148    7082 scope.go:122] "RemoveContainer" containerID="eadbbd08eb04c8129c4447170b111d1088cd459e0ed0f93ae6dc581d427cc9f8"
Oct 29 22:15:02.662653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:02.661755    7082 generic.go:356] "Generic (PLEG): container finished" podID="f7380f6d-cac7-48e0-b312-41612a211252" containerID="673e46677836cde5d3a5b812d8b6fedc54455c4cb692fa3ba8c594e5ca02d7c8" exitCode=60
Oct 29 22:15:02.662653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:02.661808    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v" event={"ID":"f7380f6d-cac7-48e0-b312-41612a211252","Type":"ContainerDied","Data":"673e46677836cde5d3a5b812d8b6fedc54455c4cb692fa3ba8c594e5ca02d7c8"}
Oct 29 22:15:02.662653 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:02.662124    7082 scope.go:122] "RemoveContainer" containerID="673e46677836cde5d3a5b812d8b6fedc54455c4cb692fa3ba8c594e5ca02d7c8"
Oct 29 22:15:03.129468 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:03.129421    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" containerID="cri-o://bd1224e58549f8c494ec711a57d3adb091215cb2fca8e93c62e69ee1227bfe0b" gracePeriod=45
Oct 29 22:15:03.497707 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:03.488710    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:15:03.655651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:03.655014    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:15:03.689669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:03.679576    7082 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q" podUID="2eb1b9db-8f2e-448e-8b42-96025377751b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:15:03.913654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:03.892260    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252" event={"ID":"bf70914c-77a1-427d-8c85-903954bde9dd","Type":"ContainerStarted","Data":"c8ccc47c82d54e5b6df82ab64b37f968836c51f667a1cc80225d575fd94eba6f"}
Oct 29 22:15:03.935654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:03.928041    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v" event={"ID":"f7380f6d-cac7-48e0-b312-41612a211252","Type":"ContainerStarted","Data":"af63483c6f51551785416228b2b83eda7e8484127c126cd2f7906a80607ead7c"}
Oct 29 22:15:04.400855 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:04.399806    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:15:04.464746 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:04.403700    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:15:04.492679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:04.471844    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": EOF"
Oct 29 22:15:04.492679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:04.472232    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:15:04.492679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:04.472655    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main" probeResult="failure" output="Get \"https://10.134.0.114:8000/ready\": dial tcp 10.134.0.114:8000: connect: connection refused"
Oct 29 22:15:04.988672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:04.988345    7082 generic.go:356] "Generic (PLEG): container finished" podID="bf70914c-77a1-427d-8c85-903954bde9dd" containerID="c8ccc47c82d54e5b6df82ab64b37f968836c51f667a1cc80225d575fd94eba6f" exitCode=60
Oct 29 22:15:04.988672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:04.988459    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252" event={"ID":"bf70914c-77a1-427d-8c85-903954bde9dd","Type":"ContainerDied","Data":"c8ccc47c82d54e5b6df82ab64b37f968836c51f667a1cc80225d575fd94eba6f"}
Oct 29 22:15:04.988672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:04.988520    7082 scope.go:122] "RemoveContainer" containerID="eadbbd08eb04c8129c4447170b111d1088cd459e0ed0f93ae6dc581d427cc9f8"
Oct 29 22:15:04.989339 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:04.989070    7082 scope.go:122] "RemoveContainer" containerID="c8ccc47c82d54e5b6df82ab64b37f968836c51f667a1cc80225d575fd94eba6f"
Oct 29 22:15:04.989478 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:04.989449    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252_odh-ai-gateway-infra(bf70914c-77a1-427d-8c85-903954bde9dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252" podUID="bf70914c-77a1-427d-8c85-903954bde9dd"
Oct 29 22:15:04.993658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:04.992998    7082 generic.go:356] "Generic (PLEG): container finished" podID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerID="bd1224e58549f8c494ec711a57d3adb091215cb2fca8e93c62e69ee1227bfe0b" exitCode=0
Oct 29 22:15:04.993658 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:04.993104    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" event={"ID":"f58ce73d-06fe-4f24-ada0-ba74bf58cb21","Type":"ContainerDied","Data":"bd1224e58549f8c494ec711a57d3adb091215cb2fca8e93c62e69ee1227bfe0b"}
Oct 29 22:15:04.998465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:04.997838    7082 generic.go:356] "Generic (PLEG): container finished" podID="f7380f6d-cac7-48e0-b312-41612a211252" containerID="af63483c6f51551785416228b2b83eda7e8484127c126cd2f7906a80607ead7c" exitCode=60
Oct 29 22:15:04.998465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:04.998059    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v" event={"ID":"f7380f6d-cac7-48e0-b312-41612a211252","Type":"ContainerDied","Data":"af63483c6f51551785416228b2b83eda7e8484127c126cd2f7906a80607ead7c"}
Oct 29 22:15:05.001368 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.000879    7082 scope.go:122] "RemoveContainer" containerID="af63483c6f51551785416228b2b83eda7e8484127c126cd2f7906a80607ead7c"
Oct 29 22:15:05.001368 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:05.001309    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v_odh-ai-gateway-infra(f7380f6d-cac7-48e0-b312-41612a211252)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v" podUID="f7380f6d-cac7-48e0-b312-41612a211252"
Oct 29 22:15:05.078018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.061851    7082 scope.go:122] "RemoveContainer" containerID="673e46677836cde5d3a5b812d8b6fedc54455c4cb692fa3ba8c594e5ca02d7c8"
Oct 29 22:15:05.170247 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.170198    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:15:05.339996 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.339965    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-home\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-home\") pod \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") "
Oct 29 22:15:05.340190 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.340023    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-kube-api-access-4mbwn\" (UniqueName: \"kubernetes.io/projected/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-kube-api-access-4mbwn\") pod \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") "
Oct 29 22:15:05.340190 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.340137    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tmp-dir\") pod \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") "
Oct 29 22:15:05.340306 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.340247    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tls-certs\" (UniqueName: \"kubernetes.io/secret/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tls-certs\") pod \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") "
Oct 29 22:15:05.340306 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.340276    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-home" pod "f58ce73d-06fe-4f24-ada0-ba74bf58cb21" (UID: "f58ce73d-06fe-4f24-ada0-ba74bf58cb21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:05.340306 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.340287    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-dshm\") pod \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") "
Oct 29 22:15:05.340477 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.340357    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-model-cache\") pod \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\" (UID: \"f58ce73d-06fe-4f24-ada0-ba74bf58cb21\") "
Oct 29 22:15:05.340537 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.340473    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tmp-dir" pod "f58ce73d-06fe-4f24-ada0-ba74bf58cb21" (UID: "f58ce73d-06fe-4f24-ada0-ba74bf58cb21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:05.340647 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.340604    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-model-cache" pod "f58ce73d-06fe-4f24-ada0-ba74bf58cb21" (UID: "f58ce73d-06fe-4f24-ada0-ba74bf58cb21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:05.340974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.340952    7082 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:05.340974 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.340975    7082 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:05.341144 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.340989    7082 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:05.342571 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.342546    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-dshm" pod "f58ce73d-06fe-4f24-ada0-ba74bf58cb21" (UID: "f58ce73d-06fe-4f24-ada0-ba74bf58cb21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:05.342571 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.342558    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tls-certs" pod "f58ce73d-06fe-4f24-ada0-ba74bf58cb21" (UID: "f58ce73d-06fe-4f24-ada0-ba74bf58cb21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:05.342713 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.342608    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-kube-api-access-4mbwn" pod "f58ce73d-06fe-4f24-ada0-ba74bf58cb21" (UID: "f58ce73d-06fe-4f24-ada0-ba74bf58cb21"). InnerVolumeSpecName "kube-api-access-4mbwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:05.441987 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.441898    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mbwn\" (UniqueName: \"kubernetes.io/projected/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-kube-api-access-4mbwn\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:05.441987 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.441932    7082 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:05.441987 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:05.441947    7082 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58ce73d-06fe-4f24-ada0-ba74bf58cb21-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:06.014442 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:06.014398    7082 scope.go:122] "RemoveContainer" containerID="c8ccc47c82d54e5b6df82ab64b37f968836c51f667a1cc80225d575fd94eba6f"
Oct 29 22:15:06.014980 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:06.014789    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252_odh-ai-gateway-infra(bf70914c-77a1-427d-8c85-903954bde9dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252" podUID="bf70914c-77a1-427d-8c85-903954bde9dd"
Oct 29 22:15:06.015787 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:06.015764    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"
Oct 29 22:15:06.015905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:06.015802    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7" event={"ID":"f58ce73d-06fe-4f24-ada0-ba74bf58cb21","Type":"ContainerDied","Data":"3bc1321e458c92e99294aec2180e36e35103466a7d6da693db89c605f0e81b0a"}
Oct 29 22:15:06.015905 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:06.015852    7082 scope.go:122] "RemoveContainer" containerID="bd1224e58549f8c494ec711a57d3adb091215cb2fca8e93c62e69ee1227bfe0b"
Oct 29 22:15:06.018353 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:06.018336    7082 scope.go:122] "RemoveContainer" containerID="af63483c6f51551785416228b2b83eda7e8484127c126cd2f7906a80607ead7c"
Oct 29 22:15:06.018586 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:06.018566    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v_odh-ai-gateway-infra(f7380f6d-cac7-48e0-b312-41612a211252)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v" podUID="f7380f6d-cac7-48e0-b312-41612a211252"
Oct 29 22:15:06.091456 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:06.091425    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"]
Oct 29 22:15:06.094668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:06.094616    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/distinct-w0-e4e24e-kserve-586d464fdd-bl4j7"]
Oct 29 22:15:06.774767 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:06.774731    7082 scope.go:122] "RemoveContainer" containerID="acc1540c6c27921b850c8a0335cf4c9e16dc7576fe8e6b527cb3c681cff18f35"
Oct 29 22:15:06.786421 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:06.786394    7082 scope.go:122] "RemoveContainer" containerID="c73c3f04f18696b2d83d849c15d5a55b63f4ec3b16837e52f577f521ec85cf8e"
Oct 29 22:15:07.650671 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:07.645025    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" path="/var/lib/kubelet/pods/f58ce73d-06fe-4f24-ada0-ba74bf58cb21/volumes"
Oct 29 22:15:07.657868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:07.657844    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"]
Oct 29 22:15:07.886480 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:07.886452    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"
Oct 29 22:15:08.063701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:08.055001    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf70914c-77a1-427d-8c85-903954bde9dd-kube-api-access-lvb8s\" (UniqueName: \"kubernetes.io/projected/bf70914c-77a1-427d-8c85-903954bde9dd-kube-api-access-lvb8s\") pod \"bf70914c-77a1-427d-8c85-903954bde9dd\" (UID: \"bf70914c-77a1-427d-8c85-903954bde9dd\") "
Oct 29 22:15:08.063701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:08.055223    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf70914c-77a1-427d-8c85-903954bde9dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf70914c-77a1-427d-8c85-903954bde9dd-maas-api-service-ca\") pod \"bf70914c-77a1-427d-8c85-903954bde9dd\" (UID: \"bf70914c-77a1-427d-8c85-903954bde9dd\") "
Oct 29 22:15:08.081387 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:08.081321    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf70914c-77a1-427d-8c85-903954bde9dd-maas-api-service-ca" pod "bf70914c-77a1-427d-8c85-903954bde9dd" (UID: "bf70914c-77a1-427d-8c85-903954bde9dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:08.124428 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:08.124389    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252" event={"ID":"bf70914c-77a1-427d-8c85-903954bde9dd","Type":"ContainerDied","Data":"2f23d0cb2ab91991f241db92a4ce549570638fed6f23d8a445684e8d32401cd4"}
Oct 29 22:15:08.124776 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:08.124756    7082 scope.go:122] "RemoveContainer" containerID="c8ccc47c82d54e5b6df82ab64b37f968836c51f667a1cc80225d575fd94eba6f"
Oct 29 22:15:08.125081 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:08.125065    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"
Oct 29 22:15:08.161037 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:08.160992    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf70914c-77a1-427d-8c85-903954bde9dd-kube-api-access-lvb8s" pod "bf70914c-77a1-427d-8c85-903954bde9dd" (UID: "bf70914c-77a1-427d-8c85-903954bde9dd"). InnerVolumeSpecName "kube-api-access-lvb8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:08.170805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:08.170771    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvb8s\" (UniqueName: \"kubernetes.io/projected/bf70914c-77a1-427d-8c85-903954bde9dd-kube-api-access-lvb8s\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:08.170805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:08.170804    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf70914c-77a1-427d-8c85-903954bde9dd-maas-api-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:08.382616 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:08.382523    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"]
Oct 29 22:15:08.385904 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:08.385875    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e4e24e-44vrj"]
Oct 29 22:15:08.494035 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:08.493950    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"]
Oct 29 22:15:08.497725 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:08.497696    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e4e24e-29888535-rb252"]
Oct 29 22:15:09.631117 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:09.631081    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf70914c-77a1-427d-8c85-903954bde9dd" path="/var/lib/kubelet/pods/bf70914c-77a1-427d-8c85-903954bde9dd/volumes"
Oct 29 22:15:09.631550 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:09.631534    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69cbc6f-03a9-4469-bfcb-718296f3f8bd" path="/var/lib/kubelet/pods/f69cbc6f-03a9-4469-bfcb-718296f3f8bd/volumes"
Oct 29 22:15:15.922816 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:15.922778    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"]
Oct 29 22:15:15.923668 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:15.923576    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy" containerID="cri-o://24ec1b352aef15539c438520cbd7f48b4672f3a51332a9e8e1708e57d01fbddf" gracePeriod=30
Oct 29 22:15:15.937129 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:15.937099    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"]
Oct 29 22:15:15.937380 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:15.937334    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" containerID="cri-o://0f90bf532e3c9e6d58a97f33019097792de223d14de04a57ebf4b4d9e2dfcc7e" gracePeriod=30
Oct 29 22:15:15.942261 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:15.942236    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"]
Oct 29 22:15:15.944450 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:15.944396    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy" containerID="cri-o://e2beb3fe2a4e75249e619eaa7eb02c810f660add66516c4b0265e3ba86605de3" gracePeriod=30
Oct 29 22:15:15.946212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:15.946186    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"]
Oct 29 22:15:15.946403 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:15.946379    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy" containerID="cri-o://da5771cb6b543f720e7ed14cd08e7bb7feefd16bb63e31e2ac800898bde88b0d" gracePeriod=30
Oct 29 22:15:17.631540 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:17.629046    7082 scope.go:122] "RemoveContainer" containerID="af63483c6f51551785416228b2b83eda7e8484127c126cd2f7906a80607ead7c"
Oct 29 22:15:18.251899 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:18.251805    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v" event={"ID":"f7380f6d-cac7-48e0-b312-41612a211252","Type":"ContainerStarted","Data":"57227c335914ea28eb515489b2f496ca0113a0887be490db86d4f517d7eaf66e"}
Oct 29 22:15:18.354805 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:18.354768    7082 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 29 22:15:18.355007 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:18.354824    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:15:19.261955 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:19.261923    7082 generic.go:356] "Generic (PLEG): container finished" podID="f7380f6d-cac7-48e0-b312-41612a211252" containerID="57227c335914ea28eb515489b2f496ca0113a0887be490db86d4f517d7eaf66e" exitCode=60
Oct 29 22:15:19.262378 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:19.261997    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v" event={"ID":"f7380f6d-cac7-48e0-b312-41612a211252","Type":"ContainerDied","Data":"57227c335914ea28eb515489b2f496ca0113a0887be490db86d4f517d7eaf66e"}
Oct 29 22:15:19.262378 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:19.262058    7082 scope.go:122] "RemoveContainer" containerID="af63483c6f51551785416228b2b83eda7e8484127c126cd2f7906a80607ead7c"
Oct 29 22:15:19.262378 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:19.262371    7082 scope.go:122] "RemoveContainer" containerID="57227c335914ea28eb515489b2f496ca0113a0887be490db86d4f517d7eaf66e"
Oct 29 22:15:19.262753 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:19.262728    7082 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v_odh-ai-gateway-infra(f7380f6d-cac7-48e0-b312-41612a211252)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v" podUID="f7380f6d-cac7-48e0-b312-41612a211252"
Oct 29 22:15:19.302781 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:19.302746    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"]
Oct 29 22:15:20.425090 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:20.425067    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"
Oct 29 22:15:20.536455 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:20.536367    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7380f6d-cac7-48e0-b312-41612a211252-kube-api-access-t9x5k\" (UniqueName: \"kubernetes.io/projected/f7380f6d-cac7-48e0-b312-41612a211252-kube-api-access-t9x5k\") pod \"f7380f6d-cac7-48e0-b312-41612a211252\" (UID: \"f7380f6d-cac7-48e0-b312-41612a211252\") "
Oct 29 22:15:20.536604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:20.536498    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7380f6d-cac7-48e0-b312-41612a211252-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7380f6d-cac7-48e0-b312-41612a211252-maas-api-service-ca\") pod \"f7380f6d-cac7-48e0-b312-41612a211252\" (UID: \"f7380f6d-cac7-48e0-b312-41612a211252\") "
Oct 29 22:15:20.536861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:20.536839    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7380f6d-cac7-48e0-b312-41612a211252-maas-api-service-ca" pod "f7380f6d-cac7-48e0-b312-41612a211252" (UID: "f7380f6d-cac7-48e0-b312-41612a211252"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:20.537124 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:20.537110    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7380f6d-cac7-48e0-b312-41612a211252-maas-api-service-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:20.538779 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:20.538758    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7380f6d-cac7-48e0-b312-41612a211252-kube-api-access-t9x5k" pod "f7380f6d-cac7-48e0-b312-41612a211252" (UID: "f7380f6d-cac7-48e0-b312-41612a211252"). InnerVolumeSpecName "kube-api-access-t9x5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:20.638918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:20.638889    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9x5k\" (UniqueName: \"kubernetes.io/projected/f7380f6d-cac7-48e0-b312-41612a211252-kube-api-access-t9x5k\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.235371 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.235337    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 22:15:21.312828 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.312793    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v" event={"ID":"f7380f6d-cac7-48e0-b312-41612a211252","Type":"ContainerDied","Data":"7d896549e0a9ca41a1949a3d8aa8229b8c379a354eef4a3954366b15e10ad984"}
Oct 29 22:15:21.313022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.312864    7082 scope.go:122] "RemoveContainer" containerID="57227c335914ea28eb515489b2f496ca0113a0887be490db86d4f517d7eaf66e"
Oct 29 22:15:21.313022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.312965    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"
Oct 29 22:15:21.318343 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.318248    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 22:15:21.322371 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.322258    7082 generic.go:356] "Generic (PLEG): container finished" podID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerID="e2beb3fe2a4e75249e619eaa7eb02c810f660add66516c4b0265e3ba86605de3" exitCode=0
Oct 29 22:15:21.323039 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.322927    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" event={"ID":"a251aea6-7480-407f-8e4e-2e3d3c5c99b4","Type":"ContainerDied","Data":"e2beb3fe2a4e75249e619eaa7eb02c810f660add66516c4b0265e3ba86605de3"}
Oct 29 22:15:21.325430 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.325270    7082 generic.go:356] "Generic (PLEG): container finished" podID="f4150595-96b9-4f34-a2be-fdb99526b429" containerID="24ec1b352aef15539c438520cbd7f48b4672f3a51332a9e8e1708e57d01fbddf" exitCode=0
Oct 29 22:15:21.325789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.325662    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" event={"ID":"f4150595-96b9-4f34-a2be-fdb99526b429","Type":"ContainerDied","Data":"24ec1b352aef15539c438520cbd7f48b4672f3a51332a9e8e1708e57d01fbddf"}
Oct 29 22:15:21.325789 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.325710    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s" event={"ID":"f4150595-96b9-4f34-a2be-fdb99526b429","Type":"ContainerDied","Data":"f7338b0c6ae90a1fc14965d29a129eb9ed308c906061eb797d7ff371a19c965f"}
Oct 29 22:15:21.326177 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.326071    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"
Oct 29 22:15:21.329504 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.329338    7082 generic.go:356] "Generic (PLEG): container finished" podID="dc27893e-a205-4145-9752-1e6813c3b88d" containerID="da5771cb6b543f720e7ed14cd08e7bb7feefd16bb63e31e2ac800898bde88b0d" exitCode=0
Oct 29 22:15:21.329504 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.329449    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" event={"ID":"dc27893e-a205-4145-9752-1e6813c3b88d","Type":"ContainerDied","Data":"da5771cb6b543f720e7ed14cd08e7bb7feefd16bb63e31e2ac800898bde88b0d"}
Oct 29 22:15:21.333760 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.333739    7082 generic.go:356] "Generic (PLEG): container finished" podID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerID="0f90bf532e3c9e6d58a97f33019097792de223d14de04a57ebf4b4d9e2dfcc7e" exitCode=0
Oct 29 22:15:21.333867 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.333821    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" event={"ID":"ad3feeed-2094-44d9-aad9-00d17e6bbef0","Type":"ContainerDied","Data":"0f90bf532e3c9e6d58a97f33019097792de223d14de04a57ebf4b4d9e2dfcc7e"}
Oct 29 22:15:21.334138 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.334030    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"
Oct 29 22:15:21.334327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.334309    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv" event={"ID":"ad3feeed-2094-44d9-aad9-00d17e6bbef0","Type":"ContainerDied","Data":"d597a9a6faa68fbdf4af4892f99b93b3ebb6cabc6142f987f409079b6b8462de"}
Oct 29 22:15:21.346151 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.346048    7082 scope.go:122] "RemoveContainer" containerID="24ec1b352aef15539c438520cbd7f48b4672f3a51332a9e8e1708e57d01fbddf"
Oct 29 22:15:21.353923 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.353893    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-envoy\") pod \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") "
Oct 29 22:15:21.354419 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.354388    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4150595-96b9-4f34-a2be-fdb99526b429-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4150595-96b9-4f34-a2be-fdb99526b429-istio-podinfo\") pod \"f4150595-96b9-4f34-a2be-fdb99526b429\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") "
Oct 29 22:15:21.354758 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.354705    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-certs\") pod \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") "
Oct 29 22:15:21.355200 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.355165    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-token\" (UniqueName: \"kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-token\") pod \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") "
Oct 29 22:15:21.355368 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.355352    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-socket\") pod \"f4150595-96b9-4f34-a2be-fdb99526b429\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") "
Oct 29 22:15:21.355535 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.355497    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-certs" pod "ad3feeed-2094-44d9-aad9-00d17e6bbef0" (UID: "ad3feeed-2094-44d9-aad9-00d17e6bbef0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.355918 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.355886    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-socket" pod "f4150595-96b9-4f34-a2be-fdb99526b429" (UID: "f4150595-96b9-4f34-a2be-fdb99526b429"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.356169 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.356144    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-credential-socket\") pod \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") "
Oct 29 22:15:21.356353 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.356338    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-envoy\") pod \"f4150595-96b9-4f34-a2be-fdb99526b429\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") "
Oct 29 22:15:21.356569 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.356554    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istiod-ca-cert\") pod \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") "
Oct 29 22:15:21.358327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.357801    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-kube-api-access-d8rvb\" (UniqueName: \"kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-kube-api-access-d8rvb\") pod \"f4150595-96b9-4f34-a2be-fdb99526b429\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") "
Oct 29 22:15:21.358327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.357855    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-istio-token\" (UniqueName: \"kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-istio-token\") pod \"f4150595-96b9-4f34-a2be-fdb99526b429\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") "
Oct 29 22:15:21.358327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.357880    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4150595-96b9-4f34-a2be-fdb99526b429-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4150595-96b9-4f34-a2be-fdb99526b429-istiod-ca-cert\") pod \"f4150595-96b9-4f34-a2be-fdb99526b429\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") "
Oct 29 22:15:21.358327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.357913    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-certs\") pod \"f4150595-96b9-4f34-a2be-fdb99526b429\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") "
Oct 29 22:15:21.358327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.357989    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-credential-socket\") pod \"f4150595-96b9-4f34-a2be-fdb99526b429\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") "
Oct 29 22:15:21.358327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.358037    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-socket\") pod \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") "
Oct 29 22:15:21.358327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.358089    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-podinfo\") pod \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") "
Oct 29 22:15:21.358327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.358165    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-data\") pod \"f4150595-96b9-4f34-a2be-fdb99526b429\" (UID: \"f4150595-96b9-4f34-a2be-fdb99526b429\") "
Oct 29 22:15:21.358327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.358206    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-kube-api-access-gwjdr\" (UniqueName: \"kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-kube-api-access-gwjdr\") pod \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") "
Oct 29 22:15:21.358327 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.358240    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-data\") pod \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\" (UID: \"ad3feeed-2094-44d9-aad9-00d17e6bbef0\") "
Oct 29 22:15:21.358927 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.358904    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.359033 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.358927    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.359695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.357113    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istiod-ca-cert" pod "ad3feeed-2094-44d9-aad9-00d17e6bbef0" (UID: "ad3feeed-2094-44d9-aad9-00d17e6bbef0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:21.359695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.357349    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-credential-socket" pod "ad3feeed-2094-44d9-aad9-00d17e6bbef0" (UID: "ad3feeed-2094-44d9-aad9-00d17e6bbef0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.359695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.357738    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-envoy" pod "ad3feeed-2094-44d9-aad9-00d17e6bbef0" (UID: "ad3feeed-2094-44d9-aad9-00d17e6bbef0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.359695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.359195    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-data" pod "ad3feeed-2094-44d9-aad9-00d17e6bbef0" (UID: "ad3feeed-2094-44d9-aad9-00d17e6bbef0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.359695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.359267    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-certs" pod "f4150595-96b9-4f34-a2be-fdb99526b429" (UID: "f4150595-96b9-4f34-a2be-fdb99526b429"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.359695 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.359657    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-envoy" pod "f4150595-96b9-4f34-a2be-fdb99526b429" (UID: "f4150595-96b9-4f34-a2be-fdb99526b429"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.360150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.359810    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-credential-socket" pod "f4150595-96b9-4f34-a2be-fdb99526b429" (UID: "f4150595-96b9-4f34-a2be-fdb99526b429"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.360150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.360025    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-data" pod "f4150595-96b9-4f34-a2be-fdb99526b429" (UID: "f4150595-96b9-4f34-a2be-fdb99526b429"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.360391 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.360330    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4150595-96b9-4f34-a2be-fdb99526b429-istio-podinfo" pod "f4150595-96b9-4f34-a2be-fdb99526b429" (UID: "f4150595-96b9-4f34-a2be-fdb99526b429"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:21.360681 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.360606    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4150595-96b9-4f34-a2be-fdb99526b429-istiod-ca-cert" pod "f4150595-96b9-4f34-a2be-fdb99526b429" (UID: "f4150595-96b9-4f34-a2be-fdb99526b429"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:21.361068 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.360996    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-socket" pod "ad3feeed-2094-44d9-aad9-00d17e6bbef0" (UID: "ad3feeed-2094-44d9-aad9-00d17e6bbef0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.361770 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.361746    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-token" pod "ad3feeed-2094-44d9-aad9-00d17e6bbef0" (UID: "ad3feeed-2094-44d9-aad9-00d17e6bbef0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.363496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.363164    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-istio-token" pod "f4150595-96b9-4f34-a2be-fdb99526b429" (UID: "f4150595-96b9-4f34-a2be-fdb99526b429"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.365274 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.365243    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-kube-api-access-d8rvb" pod "f4150595-96b9-4f34-a2be-fdb99526b429" (UID: "f4150595-96b9-4f34-a2be-fdb99526b429"). InnerVolumeSpecName "kube-api-access-d8rvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.366472 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.366048    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-kube-api-access-gwjdr" pod "ad3feeed-2094-44d9-aad9-00d17e6bbef0" (UID: "ad3feeed-2094-44d9-aad9-00d17e6bbef0"). InnerVolumeSpecName "kube-api-access-gwjdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.367042 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.366746    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-podinfo" pod "ad3feeed-2094-44d9-aad9-00d17e6bbef0" (UID: "ad3feeed-2094-44d9-aad9-00d17e6bbef0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:21.367042 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.366783    7082 scope.go:122] "RemoveContainer" containerID="24ec1b352aef15539c438520cbd7f48b4672f3a51332a9e8e1708e57d01fbddf"
Oct 29 22:15:21.367164 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:21.367102    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ec1b352aef15539c438520cbd7f48b4672f3a51332a9e8e1708e57d01fbddf\": container with ID starting with 24ec1b352aef15539c438520cbd7f48b4672f3a51332a9e8e1708e57d01fbddf not found: ID does not exist" containerID="24ec1b352aef15539c438520cbd7f48b4672f3a51332a9e8e1708e57d01fbddf"
Oct 29 22:15:21.367164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.367134    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ec1b352aef15539c438520cbd7f48b4672f3a51332a9e8e1708e57d01fbddf"} err="failed to get container status \"24ec1b352aef15539c438520cbd7f48b4672f3a51332a9e8e1708e57d01fbddf\": rpc error: code = NotFound desc = could not find container \"24ec1b352aef15539c438520cbd7f48b4672f3a51332a9e8e1708e57d01fbddf\": container with ID starting with 24ec1b352aef15539c438520cbd7f48b4672f3a51332a9e8e1708e57d01fbddf not found: ID does not exist"
Oct 29 22:15:21.367164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.367160    7082 scope.go:122] "RemoveContainer" containerID="0f90bf532e3c9e6d58a97f33019097792de223d14de04a57ebf4b4d9e2dfcc7e"
Oct 29 22:15:21.373019 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.372999    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 22:15:21.385113 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.385089    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 22:15:21.391673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.391618    7082 scope.go:122] "RemoveContainer" containerID="fc03b2c343d00572920143694606e317b96698aae6680b9a769738f78870d558"
Oct 29 22:15:21.445644 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.445601    7082 scope.go:122] "RemoveContainer" containerID="0f90bf532e3c9e6d58a97f33019097792de223d14de04a57ebf4b4d9e2dfcc7e"
Oct 29 22:15:21.446015 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:21.445988    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f90bf532e3c9e6d58a97f33019097792de223d14de04a57ebf4b4d9e2dfcc7e\": container with ID starting with 0f90bf532e3c9e6d58a97f33019097792de223d14de04a57ebf4b4d9e2dfcc7e not found: ID does not exist" containerID="0f90bf532e3c9e6d58a97f33019097792de223d14de04a57ebf4b4d9e2dfcc7e"
Oct 29 22:15:21.446092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.446025    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f90bf532e3c9e6d58a97f33019097792de223d14de04a57ebf4b4d9e2dfcc7e"} err="failed to get container status \"0f90bf532e3c9e6d58a97f33019097792de223d14de04a57ebf4b4d9e2dfcc7e\": rpc error: code = NotFound desc = could not find container \"0f90bf532e3c9e6d58a97f33019097792de223d14de04a57ebf4b4d9e2dfcc7e\": container with ID starting with 0f90bf532e3c9e6d58a97f33019097792de223d14de04a57ebf4b4d9e2dfcc7e not found: ID does not exist"
Oct 29 22:15:21.446092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.446053    7082 scope.go:122] "RemoveContainer" containerID="fc03b2c343d00572920143694606e317b96698aae6680b9a769738f78870d558"
Oct 29 22:15:21.446403 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:21.446385    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc03b2c343d00572920143694606e317b96698aae6680b9a769738f78870d558\": container with ID starting with fc03b2c343d00572920143694606e317b96698aae6680b9a769738f78870d558 not found: ID does not exist" containerID="fc03b2c343d00572920143694606e317b96698aae6680b9a769738f78870d558"
Oct 29 22:15:21.446475 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.446410    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc03b2c343d00572920143694606e317b96698aae6680b9a769738f78870d558"} err="failed to get container status \"fc03b2c343d00572920143694606e317b96698aae6680b9a769738f78870d558\": rpc error: code = NotFound desc = could not find container \"fc03b2c343d00572920143694606e317b96698aae6680b9a769738f78870d558\": container with ID starting with fc03b2c343d00572920143694606e317b96698aae6680b9a769738f78870d558 not found: ID does not exist"
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460045    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-token\" (UniqueName: \"kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-token\") pod \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460119    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-istio-token\" (UniqueName: \"kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-istio-token\") pod \"dc27893e-a205-4145-9752-1e6813c3b88d\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460153    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-socket\") pod \"dc27893e-a205-4145-9752-1e6813c3b88d\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460216    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc27893e-a205-4145-9752-1e6813c3b88d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc27893e-a205-4145-9752-1e6813c3b88d-istiod-ca-cert\") pod \"dc27893e-a205-4145-9752-1e6813c3b88d\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460249    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-envoy\") pod \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460277    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-envoy\") pod \"dc27893e-a205-4145-9752-1e6813c3b88d\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460343    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istiod-ca-cert\") pod \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460369    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-podinfo\") pod \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460445    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-data\") pod \"dc27893e-a205-4145-9752-1e6813c3b88d\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460505    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-certs\") pod \"dc27893e-a205-4145-9752-1e6813c3b88d\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460563    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-kube-api-access-llrnf\" (UniqueName: \"kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-kube-api-access-llrnf\") pod \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460597    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-credential-socket\") pod \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460684    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-data\") pod \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460746    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-credential-socket\") pod \"dc27893e-a205-4145-9752-1e6813c3b88d\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460783    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-socket\") pod \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460825    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc27893e-a205-4145-9752-1e6813c3b88d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc27893e-a205-4145-9752-1e6813c3b88d-istio-podinfo\") pod \"dc27893e-a205-4145-9752-1e6813c3b88d\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460856    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-certs\") pod \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\" (UID: \"a251aea6-7480-407f-8e4e-2e3d3c5c99b4\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460907    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-kube-api-access-zvzjw\" (UniqueName: \"kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-kube-api-access-zvzjw\") pod \"dc27893e-a205-4145-9752-1e6813c3b88d\" (UID: \"dc27893e-a205-4145-9752-1e6813c3b88d\") "
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.460965    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc27893e-a205-4145-9752-1e6813c3b88d-istiod-ca-cert" pod "dc27893e-a205-4145-9752-1e6813c3b88d" (UID: "dc27893e-a205-4145-9752-1e6813c3b88d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461237    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-socket" pod "dc27893e-a205-4145-9752-1e6813c3b88d" (UID: "dc27893e-a205-4145-9752-1e6813c3b88d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461527    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8rvb\" (UniqueName: \"kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-kube-api-access-d8rvb\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461549    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4150595-96b9-4f34-a2be-fdb99526b429-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461563    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4150595-96b9-4f34-a2be-fdb99526b429-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461577    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461590    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461602    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461615    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461670    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc27893e-a205-4145-9752-1e6813c3b88d-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461684    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461697    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461709    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwjdr\" (UniqueName: \"kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-kube-api-access-gwjdr\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461725    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461738    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461750    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4150595-96b9-4f34-a2be-fdb99526b429-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461764    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461776    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3feeed-2094-44d9-aad9-00d17e6bbef0-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461790    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4150595-96b9-4f34-a2be-fdb99526b429-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461791    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-data" pod "dc27893e-a205-4145-9752-1e6813c3b88d" (UID: "dc27893e-a205-4145-9752-1e6813c3b88d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.461802    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3feeed-2094-44d9-aad9-00d17e6bbef0-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.462116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.462013    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-socket" pod "a251aea6-7480-407f-8e4e-2e3d3c5c99b4" (UID: "a251aea6-7480-407f-8e4e-2e3d3c5c99b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.464412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.462196    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-credential-socket" pod "a251aea6-7480-407f-8e4e-2e3d3c5c99b4" (UID: "a251aea6-7480-407f-8e4e-2e3d3c5c99b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.464412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.462377    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-data" pod "a251aea6-7480-407f-8e4e-2e3d3c5c99b4" (UID: "a251aea6-7480-407f-8e4e-2e3d3c5c99b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.464412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.462557    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-credential-socket" pod "dc27893e-a205-4145-9752-1e6813c3b88d" (UID: "dc27893e-a205-4145-9752-1e6813c3b88d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.464412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.464169    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-certs" pod "dc27893e-a205-4145-9752-1e6813c3b88d" (UID: "dc27893e-a205-4145-9752-1e6813c3b88d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.464412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.464349    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istiod-ca-cert" pod "a251aea6-7480-407f-8e4e-2e3d3c5c99b4" (UID: "a251aea6-7480-407f-8e4e-2e3d3c5c99b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:21.465563 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.465526    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-certs" pod "a251aea6-7480-407f-8e4e-2e3d3c5c99b4" (UID: "a251aea6-7480-407f-8e4e-2e3d3c5c99b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.465943 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.465909    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-kube-api-access-llrnf" pod "a251aea6-7480-407f-8e4e-2e3d3c5c99b4" (UID: "a251aea6-7480-407f-8e4e-2e3d3c5c99b4"). InnerVolumeSpecName "kube-api-access-llrnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.466709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.466677    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-envoy" pod "dc27893e-a205-4145-9752-1e6813c3b88d" (UID: "dc27893e-a205-4145-9752-1e6813c3b88d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.466709 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.466685    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-envoy" pod "a251aea6-7480-407f-8e4e-2e3d3c5c99b4" (UID: "a251aea6-7480-407f-8e4e-2e3d3c5c99b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.468031 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.468010    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-token" pod "a251aea6-7480-407f-8e4e-2e3d3c5c99b4" (UID: "a251aea6-7480-407f-8e4e-2e3d3c5c99b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.468149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.468040    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-istio-token" pod "dc27893e-a205-4145-9752-1e6813c3b88d" (UID: "dc27893e-a205-4145-9752-1e6813c3b88d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.468809 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.468748    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-podinfo" pod "a251aea6-7480-407f-8e4e-2e3d3c5c99b4" (UID: "a251aea6-7480-407f-8e4e-2e3d3c5c99b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:21.469082 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.469036    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc27893e-a205-4145-9752-1e6813c3b88d-istio-podinfo" pod "dc27893e-a205-4145-9752-1e6813c3b88d" (UID: "dc27893e-a205-4145-9752-1e6813c3b88d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:21.469547 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.469527    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-kube-api-access-zvzjw" pod "dc27893e-a205-4145-9752-1e6813c3b88d" (UID: "dc27893e-a205-4145-9752-1e6813c3b88d"). InnerVolumeSpecName "kube-api-access-zvzjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.542715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.542680    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"]
Oct 29 22:15:21.546906 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.546882    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a9e5f-29888535-zrb7v"]
Oct 29 22:15:21.563512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563486    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvzjw\" (UniqueName: \"kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-kube-api-access-zvzjw\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563511    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563521    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc27893e-a205-4145-9752-1e6813c3b88d-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563530    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563537    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563547    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563554    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563561    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563568    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563575    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llrnf\" (UniqueName: \"kubernetes.io/projected/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-kube-api-access-llrnf\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563583    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563591    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563601    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc27893e-a205-4145-9752-1e6813c3b88d-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563610    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563619    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc27893e-a205-4145-9752-1e6813c3b88d-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.563679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.563662    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a251aea6-7480-407f-8e4e-2e3d3c5c99b4-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.636710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.636592    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7380f6d-cac7-48e0-b312-41612a211252" path="/var/lib/kubelet/pods/f7380f6d-cac7-48e0-b312-41612a211252/volumes"
Oct 29 22:15:21.673639 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.673601    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"]
Oct 29 22:15:21.678097 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.678075    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-jw92s"]
Oct 29 22:15:21.711872 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.711841    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"]
Oct 29 22:15:21.719790 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:21.719721    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-42gbv"]
Oct 29 22:15:22.343300 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:22.343272    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"
Oct 29 22:15:22.343466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:22.343271    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c" event={"ID":"dc27893e-a205-4145-9752-1e6813c3b88d","Type":"ContainerDied","Data":"e77bfc4ead9334d6f123b6c994b5f22de777e85f3ace2e88025c2661f1fcf30b"}
Oct 29 22:15:22.343466 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:22.343450    7082 scope.go:122] "RemoveContainer" containerID="da5771cb6b543f720e7ed14cd08e7bb7feefd16bb63e31e2ac800898bde88b0d"
Oct 29 22:15:22.347309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:22.347289    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"
Oct 29 22:15:22.347309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:22.347293    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk" event={"ID":"a251aea6-7480-407f-8e4e-2e3d3c5c99b4","Type":"ContainerDied","Data":"2d210a3f414e3e7fa518b67665efd773d856685dca49791f578c9508f9984720"}
Oct 29 22:15:22.356521 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:22.356501    7082 scope.go:122] "RemoveContainer" containerID="e2beb3fe2a4e75249e619eaa7eb02c810f660add66516c4b0265e3ba86605de3"
Oct 29 22:15:22.394320 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:22.394291    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"]
Oct 29 22:15:22.405675 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:22.405647    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-w675c"]
Oct 29 22:15:22.429202 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:22.429178    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"]
Oct 29 22:15:22.432716 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:22.432689    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-25bqk"]
Oct 29 22:15:23.632697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:23.632659    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" path="/var/lib/kubelet/pods/a251aea6-7480-407f-8e4e-2e3d3c5c99b4/volumes"
Oct 29 22:15:23.633317 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:23.633289    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" path="/var/lib/kubelet/pods/ad3feeed-2094-44d9-aad9-00d17e6bbef0/volumes"
Oct 29 22:15:23.633833 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:23.633817    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" path="/var/lib/kubelet/pods/dc27893e-a205-4145-9752-1e6813c3b88d/volumes"
Oct 29 22:15:23.634612 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:23.634595    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" path="/var/lib/kubelet/pods/f4150595-96b9-4f34-a2be-fdb99526b429/volumes"
Oct 29 22:15:28.423808 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:28.423774    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:15:37.583720 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:37.583689    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:15:40.837900 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:40.837863    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"]
Oct 29 22:15:40.838870 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:40.838807    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerName="istio-proxy" containerID="cri-o://27e45bf0b05cea3c60d77a9b6d0b2b07080776303d0407c3630995d67f70b061" gracePeriod=30
Oct 29 22:15:40.844212 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:40.844185    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"]
Oct 29 22:15:40.844544 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:40.844518    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy" containerID="cri-o://0e6fc54baaf79a5ddfd3a4086a62d24c0a23a388bae038e8665579a235bcd021" gracePeriod=30
Oct 29 22:15:40.849574 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:40.849542    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"]
Oct 29 22:15:40.849940 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:40.849898    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy" containerID="cri-o://75d6369bc74dbba4d07d9a55011e9af9f95fd2df944899d1b2c8908597ad7eec" gracePeriod=30
Oct 29 22:15:40.858654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:40.858594    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"]
Oct 29 22:15:40.858868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:40.858854    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"]
Oct 29 22:15:40.859167 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:40.859134    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy" containerID="cri-o://82a5b751361e0a22c78cefb1ab9fd7766600fab043101418f6bb9810dd694eea" gracePeriod=30
Oct 29 22:15:40.859516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:40.859479    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy" containerID="cri-o://d2edb2bf6d0af4a306c232ab7eb5ccc9e94c6a6854bec0f54b5233e6b40ce579" gracePeriod=30
Oct 29 22:15:41.283307 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283221    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"]
Oct 29 22:15:41.283786 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283772    7082 cpu_manager.go:414] "containerMap: removing container" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy"
Oct 29 22:15:41.283839 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283788    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy"
Oct 29 22:15:41.283839 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283802    7082 cpu_manager.go:414] "containerMap: removing container" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy"
Oct 29 22:15:41.283839 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283808    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy"
Oct 29 22:15:41.283839 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283824    7082 cpu_manager.go:414] "containerMap: removing container" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy"
Oct 29 22:15:41.283839 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283829    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy"
Oct 29 22:15:41.283839 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283839    7082 cpu_manager.go:414] "containerMap: removing container" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283844    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283850    7082 cpu_manager.go:414] "containerMap: removing container" podUID="f7380f6d-cac7-48e0-b312-41612a211252" containerName="cleanup"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283854    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7380f6d-cac7-48e0-b312-41612a211252" containerName="cleanup"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283867    7082 cpu_manager.go:414] "containerMap: removing container" podUID="f7380f6d-cac7-48e0-b312-41612a211252" containerName="cleanup"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283871    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7380f6d-cac7-48e0-b312-41612a211252" containerName="cleanup"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283878    7082 cpu_manager.go:414] "containerMap: removing container" podUID="bf70914c-77a1-427d-8c85-903954bde9dd" containerName="cleanup"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283882    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf70914c-77a1-427d-8c85-903954bde9dd" containerName="cleanup"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283899    7082 cpu_manager.go:414] "containerMap: removing container" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283904    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283916    7082 cpu_manager.go:414] "containerMap: removing container" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283920    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.283992    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58ce73d-06fe-4f24-ada0-ba74bf58cb21" containerName="main"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.284001    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc27893e-a205-4145-9752-1e6813c3b88d" containerName="istio-proxy"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.284007    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.284014    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3feeed-2094-44d9-aad9-00d17e6bbef0" containerName="istio-proxy"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.284022    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="a251aea6-7480-407f-8e4e-2e3d3c5c99b4" containerName="istio-proxy"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.284028    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4150595-96b9-4f34-a2be-fdb99526b429" containerName="istio-proxy"
Oct 29 22:15:41.284028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.284034    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7380f6d-cac7-48e0-b312-41612a211252" containerName="cleanup"
Oct 29 22:15:41.284654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.284040    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7380f6d-cac7-48e0-b312-41612a211252" containerName="cleanup"
Oct 29 22:15:41.284654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.284045    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf70914c-77a1-427d-8c85-903954bde9dd" containerName="cleanup"
Oct 29 22:15:41.284654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.284051    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf70914c-77a1-427d-8c85-903954bde9dd" containerName="cleanup"
Oct 29 22:15:41.284654 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.284058    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7380f6d-cac7-48e0-b312-41612a211252" containerName="cleanup"
Oct 29 22:15:41.287112 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.287096    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"
Oct 29 22:15:41.301502 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.301478    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:15:41.301646 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.301528    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:15:41.309023 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.309002    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"]
Oct 29 22:15:41.369751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.369717    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbwh\" (UniqueName: \"kubernetes.io/projected/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-kube-api-access-6nbwh\") pod \"maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf\" (UID: \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"
Oct 29 22:15:41.369951 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.369771    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf\" (UID: \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"
Oct 29 22:15:41.370036 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.370002    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf\" (UID: \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"
Oct 29 22:15:41.387955 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.387928    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:15:41.388208 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.388188    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:15:41.470910 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.470883    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf\" (UID: \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"
Oct 29 22:15:41.471066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.470935    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbwh\" (UniqueName: \"kubernetes.io/projected/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-kube-api-access-6nbwh\") pod \"maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf\" (UID: \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"
Oct 29 22:15:41.471066 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.470975    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf\" (UID: \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"
Oct 29 22:15:41.471607 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.471585    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf\" (UID: \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"
Oct 29 22:15:41.473555 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.473533    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf\" (UID: \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"
Oct 29 22:15:41.496849 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.496828    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbwh\" (UniqueName: \"kubernetes.io/projected/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-kube-api-access-6nbwh\") pod \"maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf\" (UID: \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"
Oct 29 22:15:41.597683 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.597590    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:15:41.597952 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.597720    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:15:41.600838 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.600812    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"
Oct 29 22:15:41.914850 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:41.914823    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"]
Oct 29 22:15:41.917609 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:15:41.917582    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe386d66_4289_4d0d_88f5_ae2bb63bc06c.slice/crio-527792f6a7a3ab18a74d4511fb5dc35f7936184bfd83ec41484b015142984bff WatchSource:0}: Error finding container 527792f6a7a3ab18a74d4511fb5dc35f7936184bfd83ec41484b015142984bff: Status 404 returned error can't find the container with id 527792f6a7a3ab18a74d4511fb5dc35f7936184bfd83ec41484b015142984bff
Oct 29 22:15:42.538911 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:42.538834    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf" event={"ID":"fe386d66-4289-4d0d-88f5-ae2bb63bc06c","Type":"ContainerStarted","Data":"62c651b523ad46c95870bcf1b4cf1cb9ce28340f708484c70e2e6899bb930c93"}
Oct 29 22:15:42.538911 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:42.538885    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf" event={"ID":"fe386d66-4289-4d0d-88f5-ae2bb63bc06c","Type":"ContainerStarted","Data":"527792f6a7a3ab18a74d4511fb5dc35f7936184bfd83ec41484b015142984bff"}
Oct 29 22:15:42.581479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:42.581430    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf" podStartSLOduration=1.5814118769999999 podStartE2EDuration="1.581411877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:42.578586561 +0000 UTC m=+1715.078628863" watchObservedRunningTime="2026-10-29 22:15:42.581411877 +0000 UTC m=+1715.081454176"
Oct 29 22:15:43.566962 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:43.566926    7082 generic.go:356] "Generic (PLEG): container finished" podID="fe386d66-4289-4d0d-88f5-ae2bb63bc06c" containerID="62c651b523ad46c95870bcf1b4cf1cb9ce28340f708484c70e2e6899bb930c93" exitCode=0
Oct 29 22:15:43.567503 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:43.567004    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf" event={"ID":"fe386d66-4289-4d0d-88f5-ae2bb63bc06c","Type":"ContainerDied","Data":"62c651b523ad46c95870bcf1b4cf1cb9ce28340f708484c70e2e6899bb930c93"}
Oct 29 22:15:43.637087 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:43.637052    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:15:43.637360 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:43.637331    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:15:43.637494 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:43.637185    7082 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:15:43.637494 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:43.637385    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:15:44.741617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:44.741586    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"
Oct 29 22:15:44.813939 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:44.813906    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-xks-ca\") pod \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\" (UID: \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\") "
Oct 29 22:15:44.814085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:44.813961    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-openshift-ca\") pod \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\" (UID: \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\") "
Oct 29 22:15:44.814085 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:44.814013    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-kube-api-access-6nbwh\" (UniqueName: \"kubernetes.io/projected/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-kube-api-access-6nbwh\") pod \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\" (UID: \"fe386d66-4289-4d0d-88f5-ae2bb63bc06c\") "
Oct 29 22:15:44.814407 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:44.814380    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-openshift-ca" pod "fe386d66-4289-4d0d-88f5-ae2bb63bc06c" (UID: "fe386d66-4289-4d0d-88f5-ae2bb63bc06c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:44.814935 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:44.814909    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:44.816262 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:44.816239    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-xks-ca" pod "fe386d66-4289-4d0d-88f5-ae2bb63bc06c" (UID: "fe386d66-4289-4d0d-88f5-ae2bb63bc06c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:44.816542 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:44.816520    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-kube-api-access-6nbwh" pod "fe386d66-4289-4d0d-88f5-ae2bb63bc06c" (UID: "fe386d66-4289-4d0d-88f5-ae2bb63bc06c"). InnerVolumeSpecName "kube-api-access-6nbwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:44.916227 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:44.916171    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:44.916227 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:44.916196    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nbwh\" (UniqueName: \"kubernetes.io/projected/fe386d66-4289-4d0d-88f5-ae2bb63bc06c-kube-api-access-6nbwh\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:45.583544 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:45.583513    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"
Oct 29 22:15:45.583731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:45.583540    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf" event={"ID":"fe386d66-4289-4d0d-88f5-ae2bb63bc06c","Type":"ContainerDied","Data":"527792f6a7a3ab18a74d4511fb5dc35f7936184bfd83ec41484b015142984bff"}
Oct 29 22:15:45.583731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:45.583614    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="527792f6a7a3ab18a74d4511fb5dc35f7936184bfd83ec41484b015142984bff"
Oct 29 22:15:46.284618 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.284587    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:15:46.334351 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.334329    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:15:46.339491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.339471    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:15:46.344194 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.344126    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:15:46.345663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.344498    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-envoy\") pod \"4301312f-6325-4e8c-9df7-8e4153fbcea9\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") "
Oct 29 22:15:46.345663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.344566    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-credential-socket\") pod \"4301312f-6325-4e8c-9df7-8e4153fbcea9\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") "
Oct 29 22:15:46.345663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.344594    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-socket\") pod \"4301312f-6325-4e8c-9df7-8e4153fbcea9\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") "
Oct 29 22:15:46.345663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.344722    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-kube-api-access-wrt49\" (UniqueName: \"kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-kube-api-access-wrt49\") pod \"4301312f-6325-4e8c-9df7-8e4153fbcea9\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") "
Oct 29 22:15:46.345663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.344752    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-token\" (UniqueName: \"kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-token\") pod \"4301312f-6325-4e8c-9df7-8e4153fbcea9\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") "
Oct 29 22:15:46.345663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.344796    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4301312f-6325-4e8c-9df7-8e4153fbcea9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4301312f-6325-4e8c-9df7-8e4153fbcea9-istiod-ca-cert\") pod \"4301312f-6325-4e8c-9df7-8e4153fbcea9\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") "
Oct 29 22:15:46.345663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.344839    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-podinfo\") pod \"4301312f-6325-4e8c-9df7-8e4153fbcea9\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") "
Oct 29 22:15:46.345663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.344879    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-data\") pod \"4301312f-6325-4e8c-9df7-8e4153fbcea9\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") "
Oct 29 22:15:46.345663 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.344910    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-certs\") pod \"4301312f-6325-4e8c-9df7-8e4153fbcea9\" (UID: \"4301312f-6325-4e8c-9df7-8e4153fbcea9\") "
Oct 29 22:15:46.349346 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.345909    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-certs" pod "4301312f-6325-4e8c-9df7-8e4153fbcea9" (UID: "4301312f-6325-4e8c-9df7-8e4153fbcea9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.351977 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.351953    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4301312f-6325-4e8c-9df7-8e4153fbcea9-istiod-ca-cert" pod "4301312f-6325-4e8c-9df7-8e4153fbcea9" (UID: "4301312f-6325-4e8c-9df7-8e4153fbcea9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:46.352277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.352223    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-data" pod "4301312f-6325-4e8c-9df7-8e4153fbcea9" (UID: "4301312f-6325-4e8c-9df7-8e4153fbcea9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.352725 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.352703    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-credential-socket" pod "4301312f-6325-4e8c-9df7-8e4153fbcea9" (UID: "4301312f-6325-4e8c-9df7-8e4153fbcea9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.352883 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.352721    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-socket" pod "4301312f-6325-4e8c-9df7-8e4153fbcea9" (UID: "4301312f-6325-4e8c-9df7-8e4153fbcea9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.354297 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.354268    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-podinfo" pod "4301312f-6325-4e8c-9df7-8e4153fbcea9" (UID: "4301312f-6325-4e8c-9df7-8e4153fbcea9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:46.356238 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.356212    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-token" pod "4301312f-6325-4e8c-9df7-8e4153fbcea9" (UID: "4301312f-6325-4e8c-9df7-8e4153fbcea9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.356602 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.356581    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:15:46.356904 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.356867    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-envoy" pod "4301312f-6325-4e8c-9df7-8e4153fbcea9" (UID: "4301312f-6325-4e8c-9df7-8e4153fbcea9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.358016 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.357992    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-kube-api-access-wrt49" pod "4301312f-6325-4e8c-9df7-8e4153fbcea9" (UID: "4301312f-6325-4e8c-9df7-8e4153fbcea9"). InnerVolumeSpecName "kube-api-access-wrt49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446016    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istiod-ca-cert\") pod \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446509    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-istio-token\" (UniqueName: \"kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-istio-token\") pod \"2e69aa18-c109-4098-b08a-67695f24fa34\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446549    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-token\" (UniqueName: \"kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-token\") pod \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446437    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istiod-ca-cert" pod "2a7fa2cd-1234-4b80-b869-3ab7019d1cea" (UID: "2a7fa2cd-1234-4b80-b869-3ab7019d1cea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446575    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-credential-socket\") pod \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446606    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-data\") pod \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446659    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-envoy\") pod \"d3833557-dcfc-4ed5-8415-cca71bba35e7\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446686    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-data\") pod \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446732    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-kube-api-access-qk98d\" (UniqueName: \"kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-kube-api-access-qk98d\") pod \"2e69aa18-c109-4098-b08a-67695f24fa34\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446762    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-socket\") pod \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446802    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-certs\") pod \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446840    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-kube-api-access-mzmzt\" (UniqueName: \"kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-kube-api-access-mzmzt\") pod \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446865    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-token\" (UniqueName: \"kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-token\") pod \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446890    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-socket\") pod \"d3833557-dcfc-4ed5-8415-cca71bba35e7\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446948    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3833557-dcfc-4ed5-8415-cca71bba35e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3833557-dcfc-4ed5-8415-cca71bba35e7-istiod-ca-cert\") pod \"d3833557-dcfc-4ed5-8415-cca71bba35e7\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.446977    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-envoy\") pod \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447006    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-socket\") pod \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447046    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-kube-api-access-z6j8w\" (UniqueName: \"kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-kube-api-access-z6j8w\") pod \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447083    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-token\" (UniqueName: \"kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-token\") pod \"d3833557-dcfc-4ed5-8415-cca71bba35e7\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447115    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-certs\") pod \"d3833557-dcfc-4ed5-8415-cca71bba35e7\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447141    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-data\") pod \"2e69aa18-c109-4098-b08a-67695f24fa34\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447184    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-data\") pod \"d3833557-dcfc-4ed5-8415-cca71bba35e7\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447214    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-kube-api-access-zcbm7\" (UniqueName: \"kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-kube-api-access-zcbm7\") pod \"d3833557-dcfc-4ed5-8415-cca71bba35e7\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447397    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e69aa18-c109-4098-b08a-67695f24fa34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e69aa18-c109-4098-b08a-67695f24fa34-istio-podinfo\") pod \"2e69aa18-c109-4098-b08a-67695f24fa34\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447429    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istiod-ca-cert\") pod \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447469    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-podinfo\") pod \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\" (UID: \"2a7fa2cd-1234-4b80-b869-3ab7019d1cea\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447499    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-podinfo\") pod \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447532    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-certs\") pod \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447568    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e69aa18-c109-4098-b08a-67695f24fa34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e69aa18-c109-4098-b08a-67695f24fa34-istiod-ca-cert\") pod \"2e69aa18-c109-4098-b08a-67695f24fa34\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447694    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-credential-socket\") pod \"2e69aa18-c109-4098-b08a-67695f24fa34\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447768    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-podinfo\") pod \"d3833557-dcfc-4ed5-8415-cca71bba35e7\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447799    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-envoy\") pod \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447856    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-credential-socket\") pod \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\" (UID: \"ac2febe4-d73f-42e3-bf74-ce53ba139b78\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447932    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-certs\") pod \"2e69aa18-c109-4098-b08a-67695f24fa34\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447969    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-credential-socket\") pod \"d3833557-dcfc-4ed5-8415-cca71bba35e7\" (UID: \"d3833557-dcfc-4ed5-8415-cca71bba35e7\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.447998    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-envoy\") pod \"2e69aa18-c109-4098-b08a-67695f24fa34\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.448026    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-socket\") pod \"2e69aa18-c109-4098-b08a-67695f24fa34\" (UID: \"2e69aa18-c109-4098-b08a-67695f24fa34\") "
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.448971    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-data" pod "d3833557-dcfc-4ed5-8415-cca71bba35e7" (UID: "d3833557-dcfc-4ed5-8415-cca71bba35e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.449010    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-credential-socket" pod "ac2febe4-d73f-42e3-bf74-ce53ba139b78" (UID: "ac2febe4-d73f-42e3-bf74-ce53ba139b78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.449167    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-data" pod "ac2febe4-d73f-42e3-bf74-ce53ba139b78" (UID: "ac2febe4-d73f-42e3-bf74-ce53ba139b78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.449352 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.449321    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-istio-token" pod "2e69aa18-c109-4098-b08a-67695f24fa34" (UID: "2e69aa18-c109-4098-b08a-67695f24fa34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.452150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.449852    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-credential-socket" pod "2e69aa18-c109-4098-b08a-67695f24fa34" (UID: "2e69aa18-c109-4098-b08a-67695f24fa34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.452150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.450349    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-envoy" pod "d3833557-dcfc-4ed5-8415-cca71bba35e7" (UID: "d3833557-dcfc-4ed5-8415-cca71bba35e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.452150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.450544    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-credential-socket" pod "2a7fa2cd-1234-4b80-b869-3ab7019d1cea" (UID: "2a7fa2cd-1234-4b80-b869-3ab7019d1cea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.452150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.450617    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-data" pod "2a7fa2cd-1234-4b80-b869-3ab7019d1cea" (UID: "2a7fa2cd-1234-4b80-b869-3ab7019d1cea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.452150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.450946    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3833557-dcfc-4ed5-8415-cca71bba35e7-istiod-ca-cert" pod "d3833557-dcfc-4ed5-8415-cca71bba35e7" (UID: "d3833557-dcfc-4ed5-8415-cca71bba35e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:46.452150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.451178    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-socket" pod "2a7fa2cd-1234-4b80-b869-3ab7019d1cea" (UID: "2a7fa2cd-1234-4b80-b869-3ab7019d1cea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.452150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.451375    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-certs" pod "2a7fa2cd-1234-4b80-b869-3ab7019d1cea" (UID: "2a7fa2cd-1234-4b80-b869-3ab7019d1cea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.452150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.451991    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e69aa18-c109-4098-b08a-67695f24fa34-istiod-ca-cert" pod "2e69aa18-c109-4098-b08a-67695f24fa34" (UID: "2e69aa18-c109-4098-b08a-67695f24fa34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:46.454058 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.453923    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-credential-socket" pod "d3833557-dcfc-4ed5-8415-cca71bba35e7" (UID: "d3833557-dcfc-4ed5-8415-cca71bba35e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.454906 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.454875    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-certs" pod "2e69aa18-c109-4098-b08a-67695f24fa34" (UID: "2e69aa18-c109-4098-b08a-67695f24fa34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.455128 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455100    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-socket" pod "2e69aa18-c109-4098-b08a-67695f24fa34" (UID: "2e69aa18-c109-4098-b08a-67695f24fa34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.455210 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455134    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-kube-api-access-z6j8w" pod "ac2febe4-d73f-42e3-bf74-ce53ba139b78" (UID: "ac2febe4-d73f-42e3-bf74-ce53ba139b78"). InnerVolumeSpecName "kube-api-access-z6j8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.455210 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455196    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.455384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455218    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.455384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455236    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.455384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455250    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.455384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455266    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrt49\" (UniqueName: \"kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-kube-api-access-wrt49\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.455384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455283    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.455384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455298    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4301312f-6325-4e8c-9df7-8e4153fbcea9-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.455384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455313    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.455384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455328    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.455384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455341    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4301312f-6325-4e8c-9df7-8e4153fbcea9-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.455911 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455876    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-kube-api-access-mzmzt" pod "2a7fa2cd-1234-4b80-b869-3ab7019d1cea" (UID: "2a7fa2cd-1234-4b80-b869-3ab7019d1cea"). InnerVolumeSpecName "kube-api-access-mzmzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.455999 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.455976    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-kube-api-access-qk98d" pod "2e69aa18-c109-4098-b08a-67695f24fa34" (UID: "2e69aa18-c109-4098-b08a-67695f24fa34"). InnerVolumeSpecName "kube-api-access-qk98d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.456536 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.456516    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-socket" pod "ac2febe4-d73f-42e3-bf74-ce53ba139b78" (UID: "ac2febe4-d73f-42e3-bf74-ce53ba139b78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.456746 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.456727    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-certs" pod "d3833557-dcfc-4ed5-8415-cca71bba35e7" (UID: "d3833557-dcfc-4ed5-8415-cca71bba35e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.456929 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.456803    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-podinfo" pod "2a7fa2cd-1234-4b80-b869-3ab7019d1cea" (UID: "2a7fa2cd-1234-4b80-b869-3ab7019d1cea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:46.458761 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.458553    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-token" pod "ac2febe4-d73f-42e3-bf74-ce53ba139b78" (UID: "ac2febe4-d73f-42e3-bf74-ce53ba139b78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.458879 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.458836    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-token" pod "d3833557-dcfc-4ed5-8415-cca71bba35e7" (UID: "d3833557-dcfc-4ed5-8415-cca71bba35e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.459216 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.459001    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-socket" pod "d3833557-dcfc-4ed5-8415-cca71bba35e7" (UID: "d3833557-dcfc-4ed5-8415-cca71bba35e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.459216 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.459100    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-data" pod "2e69aa18-c109-4098-b08a-67695f24fa34" (UID: "2e69aa18-c109-4098-b08a-67695f24fa34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.459216 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.459114    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-token" pod "2a7fa2cd-1234-4b80-b869-3ab7019d1cea" (UID: "2a7fa2cd-1234-4b80-b869-3ab7019d1cea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.459604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.459465    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-podinfo" pod "d3833557-dcfc-4ed5-8415-cca71bba35e7" (UID: "d3833557-dcfc-4ed5-8415-cca71bba35e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:46.459604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.459544    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-envoy" pod "2a7fa2cd-1234-4b80-b869-3ab7019d1cea" (UID: "2a7fa2cd-1234-4b80-b869-3ab7019d1cea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.459784 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.459746    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istiod-ca-cert" pod "ac2febe4-d73f-42e3-bf74-ce53ba139b78" (UID: "ac2febe4-d73f-42e3-bf74-ce53ba139b78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:46.459972 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.459949    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-certs" pod "ac2febe4-d73f-42e3-bf74-ce53ba139b78" (UID: "ac2febe4-d73f-42e3-bf74-ce53ba139b78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.460332 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.460147    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-envoy" pod "ac2febe4-d73f-42e3-bf74-ce53ba139b78" (UID: "ac2febe4-d73f-42e3-bf74-ce53ba139b78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.460332 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.460170    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-envoy" pod "2e69aa18-c109-4098-b08a-67695f24fa34" (UID: "2e69aa18-c109-4098-b08a-67695f24fa34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.460332 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.460303    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e69aa18-c109-4098-b08a-67695f24fa34-istio-podinfo" pod "2e69aa18-c109-4098-b08a-67695f24fa34" (UID: "2e69aa18-c109-4098-b08a-67695f24fa34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:46.460848 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.460823    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-kube-api-access-zcbm7" pod "d3833557-dcfc-4ed5-8415-cca71bba35e7" (UID: "d3833557-dcfc-4ed5-8415-cca71bba35e7"). InnerVolumeSpecName "kube-api-access-zcbm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.461577 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.461553    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-podinfo" pod "ac2febe4-d73f-42e3-bf74-ce53ba139b78" (UID: "ac2febe4-d73f-42e3-bf74-ce53ba139b78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:46.556720 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556686    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.556720 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556713    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.556720 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556723    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556731    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556741    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556750    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556758    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk98d\" (UniqueName: \"kubernetes.io/projected/2e69aa18-c109-4098-b08a-67695f24fa34-kube-api-access-qk98d\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556767    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556775    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556783    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzmzt\" (UniqueName: \"kubernetes.io/projected/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-kube-api-access-mzmzt\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556791    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556821    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556829    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3833557-dcfc-4ed5-8415-cca71bba35e7-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556837    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556845    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556852    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6j8w\" (UniqueName: \"kubernetes.io/projected/ac2febe4-d73f-42e3-bf74-ce53ba139b78-kube-api-access-z6j8w\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556859    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556867    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556876    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556883    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556892    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcbm7\" (UniqueName: \"kubernetes.io/projected/d3833557-dcfc-4ed5-8415-cca71bba35e7-kube-api-access-zcbm7\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556900    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e69aa18-c109-4098-b08a-67695f24fa34-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556909    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556916    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7fa2cd-1234-4b80-b869-3ab7019d1cea-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556924    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556931    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556939    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e69aa18-c109-4098-b08a-67695f24fa34-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556947    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556956    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3833557-dcfc-4ed5-8415-cca71bba35e7-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556963    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556970    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2febe4-d73f-42e3-bf74-ce53ba139b78-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556978    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556986    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3833557-dcfc-4ed5-8415-cca71bba35e7-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.556994    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.557028 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.557001    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e69aa18-c109-4098-b08a-67695f24fa34-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.592538 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.592508    7082 generic.go:356] "Generic (PLEG): container finished" podID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerID="82a5b751361e0a22c78cefb1ab9fd7766600fab043101418f6bb9810dd694eea" exitCode=0
Oct 29 22:15:46.592702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.592569    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"
Oct 29 22:15:46.592702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.592601    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" event={"ID":"ac2febe4-d73f-42e3-bf74-ce53ba139b78","Type":"ContainerDied","Data":"82a5b751361e0a22c78cefb1ab9fd7766600fab043101418f6bb9810dd694eea"}
Oct 29 22:15:46.592702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.592685    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn" event={"ID":"ac2febe4-d73f-42e3-bf74-ce53ba139b78","Type":"ContainerDied","Data":"082a535a433e682200cba328fce42ce8e62e37a658eedc86c828a12b81f611de"}
Oct 29 22:15:46.592876 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.592718    7082 scope.go:122] "RemoveContainer" containerID="82a5b751361e0a22c78cefb1ab9fd7766600fab043101418f6bb9810dd694eea"
Oct 29 22:15:46.594348 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.594321    7082 generic.go:356] "Generic (PLEG): container finished" podID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerID="27e45bf0b05cea3c60d77a9b6d0b2b07080776303d0407c3630995d67f70b061" exitCode=0
Oct 29 22:15:46.594465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.594374    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"
Oct 29 22:15:46.594465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.594411    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" event={"ID":"d3833557-dcfc-4ed5-8415-cca71bba35e7","Type":"ContainerDied","Data":"27e45bf0b05cea3c60d77a9b6d0b2b07080776303d0407c3630995d67f70b061"}
Oct 29 22:15:46.594465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.594444    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8" event={"ID":"d3833557-dcfc-4ed5-8415-cca71bba35e7","Type":"ContainerDied","Data":"425040e89298e01aac35a9cf382c6a42e7f009bed4214f8705a0df092cfa4818"}
Oct 29 22:15:46.596525 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.596502    7082 generic.go:356] "Generic (PLEG): container finished" podID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerID="75d6369bc74dbba4d07d9a55011e9af9f95fd2df944899d1b2c8908597ad7eec" exitCode=0
Oct 29 22:15:46.596690 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.596566    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"
Oct 29 22:15:46.596690 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.596587    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" event={"ID":"2a7fa2cd-1234-4b80-b869-3ab7019d1cea","Type":"ContainerDied","Data":"75d6369bc74dbba4d07d9a55011e9af9f95fd2df944899d1b2c8908597ad7eec"}
Oct 29 22:15:46.596832 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.596708    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj" event={"ID":"2a7fa2cd-1234-4b80-b869-3ab7019d1cea","Type":"ContainerDied","Data":"02848b0ca1ef1afd56f491bb74b4504283a99d22e0e668f1af8d6df11c3b73d1"}
Oct 29 22:15:46.598106 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.598071    7082 generic.go:356] "Generic (PLEG): container finished" podID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerID="0e6fc54baaf79a5ddfd3a4086a62d24c0a23a388bae038e8665579a235bcd021" exitCode=0
Oct 29 22:15:46.598243 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.598227    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" event={"ID":"4301312f-6325-4e8c-9df7-8e4153fbcea9","Type":"ContainerDied","Data":"0e6fc54baaf79a5ddfd3a4086a62d24c0a23a388bae038e8665579a235bcd021"}
Oct 29 22:15:46.598283 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.598271    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g" event={"ID":"4301312f-6325-4e8c-9df7-8e4153fbcea9","Type":"ContainerDied","Data":"0ed37f2ddb432486002840eb644b9733e2a6271c98c5f7e9d38fbca2f0f8ba3a"}
Oct 29 22:15:46.598382 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.598347    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"
Oct 29 22:15:46.601462 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.601438    7082 generic.go:356] "Generic (PLEG): container finished" podID="2e69aa18-c109-4098-b08a-67695f24fa34" containerID="d2edb2bf6d0af4a306c232ab7eb5ccc9e94c6a6854bec0f54b5233e6b40ce579" exitCode=0
Oct 29 22:15:46.601553 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.601482    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" event={"ID":"2e69aa18-c109-4098-b08a-67695f24fa34","Type":"ContainerDied","Data":"d2edb2bf6d0af4a306c232ab7eb5ccc9e94c6a6854bec0f54b5233e6b40ce579"}
Oct 29 22:15:46.601553 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.601516    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw" event={"ID":"2e69aa18-c109-4098-b08a-67695f24fa34","Type":"ContainerDied","Data":"71b18c7b4a48218cd7acf7a51efa852d5cd3a189d55ceea73cc1c25c4f58bd26"}
Oct 29 22:15:46.601691 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.601588    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"
Oct 29 22:15:46.607457 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.606603    7082 scope.go:122] "RemoveContainer" containerID="82a5b751361e0a22c78cefb1ab9fd7766600fab043101418f6bb9810dd694eea"
Oct 29 22:15:46.607961 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:46.607930    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a5b751361e0a22c78cefb1ab9fd7766600fab043101418f6bb9810dd694eea\": container with ID starting with 82a5b751361e0a22c78cefb1ab9fd7766600fab043101418f6bb9810dd694eea not found: ID does not exist" containerID="82a5b751361e0a22c78cefb1ab9fd7766600fab043101418f6bb9810dd694eea"
Oct 29 22:15:46.608065 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.607971    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a5b751361e0a22c78cefb1ab9fd7766600fab043101418f6bb9810dd694eea"} err="failed to get container status \"82a5b751361e0a22c78cefb1ab9fd7766600fab043101418f6bb9810dd694eea\": rpc error: code = NotFound desc = could not find container \"82a5b751361e0a22c78cefb1ab9fd7766600fab043101418f6bb9810dd694eea\": container with ID starting with 82a5b751361e0a22c78cefb1ab9fd7766600fab043101418f6bb9810dd694eea not found: ID does not exist"
Oct 29 22:15:46.608065 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.607993    7082 scope.go:122] "RemoveContainer" containerID="27e45bf0b05cea3c60d77a9b6d0b2b07080776303d0407c3630995d67f70b061"
Oct 29 22:15:46.619315 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.619300    7082 scope.go:122] "RemoveContainer" containerID="27e45bf0b05cea3c60d77a9b6d0b2b07080776303d0407c3630995d67f70b061"
Oct 29 22:15:46.619560 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:46.619541    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e45bf0b05cea3c60d77a9b6d0b2b07080776303d0407c3630995d67f70b061\": container with ID starting with 27e45bf0b05cea3c60d77a9b6d0b2b07080776303d0407c3630995d67f70b061 not found: ID does not exist" containerID="27e45bf0b05cea3c60d77a9b6d0b2b07080776303d0407c3630995d67f70b061"
Oct 29 22:15:46.619664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.619565    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e45bf0b05cea3c60d77a9b6d0b2b07080776303d0407c3630995d67f70b061"} err="failed to get container status \"27e45bf0b05cea3c60d77a9b6d0b2b07080776303d0407c3630995d67f70b061\": rpc error: code = NotFound desc = could not find container \"27e45bf0b05cea3c60d77a9b6d0b2b07080776303d0407c3630995d67f70b061\": container with ID starting with 27e45bf0b05cea3c60d77a9b6d0b2b07080776303d0407c3630995d67f70b061 not found: ID does not exist"
Oct 29 22:15:46.619664 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.619579    7082 scope.go:122] "RemoveContainer" containerID="75d6369bc74dbba4d07d9a55011e9af9f95fd2df944899d1b2c8908597ad7eec"
Oct 29 22:15:46.629067 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.629035    7082 scope.go:122] "RemoveContainer" containerID="e3229d11c4b9a6fb6a918a8920bf4458b74679563245fb37b49261b3164cde8f"
Oct 29 22:15:46.641590 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.641564    7082 scope.go:122] "RemoveContainer" containerID="75d6369bc74dbba4d07d9a55011e9af9f95fd2df944899d1b2c8908597ad7eec"
Oct 29 22:15:46.641893 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:46.641872    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d6369bc74dbba4d07d9a55011e9af9f95fd2df944899d1b2c8908597ad7eec\": container with ID starting with 75d6369bc74dbba4d07d9a55011e9af9f95fd2df944899d1b2c8908597ad7eec not found: ID does not exist" containerID="75d6369bc74dbba4d07d9a55011e9af9f95fd2df944899d1b2c8908597ad7eec"
Oct 29 22:15:46.642024 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.641897    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d6369bc74dbba4d07d9a55011e9af9f95fd2df944899d1b2c8908597ad7eec"} err="failed to get container status \"75d6369bc74dbba4d07d9a55011e9af9f95fd2df944899d1b2c8908597ad7eec\": rpc error: code = NotFound desc = could not find container \"75d6369bc74dbba4d07d9a55011e9af9f95fd2df944899d1b2c8908597ad7eec\": container with ID starting with 75d6369bc74dbba4d07d9a55011e9af9f95fd2df944899d1b2c8908597ad7eec not found: ID does not exist"
Oct 29 22:15:46.642024 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.641911    7082 scope.go:122] "RemoveContainer" containerID="e3229d11c4b9a6fb6a918a8920bf4458b74679563245fb37b49261b3164cde8f"
Oct 29 22:15:46.642181 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:46.642144    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3229d11c4b9a6fb6a918a8920bf4458b74679563245fb37b49261b3164cde8f\": container with ID starting with e3229d11c4b9a6fb6a918a8920bf4458b74679563245fb37b49261b3164cde8f not found: ID does not exist" containerID="e3229d11c4b9a6fb6a918a8920bf4458b74679563245fb37b49261b3164cde8f"
Oct 29 22:15:46.642181 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.642167    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3229d11c4b9a6fb6a918a8920bf4458b74679563245fb37b49261b3164cde8f"} err="failed to get container status \"e3229d11c4b9a6fb6a918a8920bf4458b74679563245fb37b49261b3164cde8f\": rpc error: code = NotFound desc = could not find container \"e3229d11c4b9a6fb6a918a8920bf4458b74679563245fb37b49261b3164cde8f\": container with ID starting with e3229d11c4b9a6fb6a918a8920bf4458b74679563245fb37b49261b3164cde8f not found: ID does not exist"
Oct 29 22:15:46.642181 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.642180    7082 scope.go:122] "RemoveContainer" containerID="0e6fc54baaf79a5ddfd3a4086a62d24c0a23a388bae038e8665579a235bcd021"
Oct 29 22:15:46.653617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.653598    7082 scope.go:122] "RemoveContainer" containerID="0e6fc54baaf79a5ddfd3a4086a62d24c0a23a388bae038e8665579a235bcd021"
Oct 29 22:15:46.653908 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:46.653889    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e6fc54baaf79a5ddfd3a4086a62d24c0a23a388bae038e8665579a235bcd021\": container with ID starting with 0e6fc54baaf79a5ddfd3a4086a62d24c0a23a388bae038e8665579a235bcd021 not found: ID does not exist" containerID="0e6fc54baaf79a5ddfd3a4086a62d24c0a23a388bae038e8665579a235bcd021"
Oct 29 22:15:46.653985 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.653911    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e6fc54baaf79a5ddfd3a4086a62d24c0a23a388bae038e8665579a235bcd021"} err="failed to get container status \"0e6fc54baaf79a5ddfd3a4086a62d24c0a23a388bae038e8665579a235bcd021\": rpc error: code = NotFound desc = could not find container \"0e6fc54baaf79a5ddfd3a4086a62d24c0a23a388bae038e8665579a235bcd021\": container with ID starting with 0e6fc54baaf79a5ddfd3a4086a62d24c0a23a388bae038e8665579a235bcd021 not found: ID does not exist"
Oct 29 22:15:46.653985 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.653924    7082 scope.go:122] "RemoveContainer" containerID="d2edb2bf6d0af4a306c232ab7eb5ccc9e94c6a6854bec0f54b5233e6b40ce579"
Oct 29 22:15:46.663375 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.663343    7082 scope.go:122] "RemoveContainer" containerID="d2edb2bf6d0af4a306c232ab7eb5ccc9e94c6a6854bec0f54b5233e6b40ce579"
Oct 29 22:15:46.663620 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:46.663597    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2edb2bf6d0af4a306c232ab7eb5ccc9e94c6a6854bec0f54b5233e6b40ce579\": container with ID starting with d2edb2bf6d0af4a306c232ab7eb5ccc9e94c6a6854bec0f54b5233e6b40ce579 not found: ID does not exist" containerID="d2edb2bf6d0af4a306c232ab7eb5ccc9e94c6a6854bec0f54b5233e6b40ce579"
Oct 29 22:15:46.663735 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.663655    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2edb2bf6d0af4a306c232ab7eb5ccc9e94c6a6854bec0f54b5233e6b40ce579"} err="failed to get container status \"d2edb2bf6d0af4a306c232ab7eb5ccc9e94c6a6854bec0f54b5233e6b40ce579\": rpc error: code = NotFound desc = could not find container \"d2edb2bf6d0af4a306c232ab7eb5ccc9e94c6a6854bec0f54b5233e6b40ce579\": container with ID starting with d2edb2bf6d0af4a306c232ab7eb5ccc9e94c6a6854bec0f54b5233e6b40ce579 not found: ID does not exist"
Oct 29 22:15:46.685837 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.685806    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"]
Oct 29 22:15:46.689277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.689258    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-765hn"]
Oct 29 22:15:46.735672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.735609    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"]
Oct 29 22:15:46.739184 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.739159    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-8ws8g"]
Oct 29 22:15:46.771497 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.771476    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"]
Oct 29 22:15:46.775169 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.775150    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5ctw"]
Oct 29 22:15:46.808119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.808097    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"]
Oct 29 22:15:46.812003 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.811984    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-zbrz8"]
Oct 29 22:15:46.845756 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.845733    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"]
Oct 29 22:15:46.851199 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:46.851171    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-g5mvj"]
Oct 29 22:15:47.631189 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:47.631148    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" path="/var/lib/kubelet/pods/2a7fa2cd-1234-4b80-b869-3ab7019d1cea/volumes"
Oct 29 22:15:47.631812 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:47.631792    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" path="/var/lib/kubelet/pods/2e69aa18-c109-4098-b08a-67695f24fa34/volumes"
Oct 29 22:15:47.632341 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:47.632266    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" path="/var/lib/kubelet/pods/4301312f-6325-4e8c-9df7-8e4153fbcea9/volumes"
Oct 29 22:15:47.633150 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:47.633131    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" path="/var/lib/kubelet/pods/ac2febe4-d73f-42e3-bf74-ce53ba139b78/volumes"
Oct 29 22:15:47.633767 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:47.633747    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" path="/var/lib/kubelet/pods/d3833557-dcfc-4ed5-8415-cca71bba35e7/volumes"
Oct 29 22:15:55.420553 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.420515    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"]
Oct 29 22:15:55.421062 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.420773    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv" podUID="def131f2-c87a-4d18-9ad3-144bfc1d7f67" containerName="payload-processing" containerID="cri-o://198d2a831be01ae3b0504c3cca592fae36c71a817541fae58b9a6d793078c663" gracePeriod=30
Oct 29 22:15:55.449678 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.449651    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"]
Oct 29 22:15:55.449901 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.449874    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7" podUID="b1a26227-4e48-4b02-8b28-d4eaadc78c80" containerName="payload-pre-processing" containerID="cri-o://0760a8dcae0649dcbc31a4124f6007d9b95b5805e426c348e2908b36eacfccb5" gracePeriod=30
Oct 29 22:15:55.703903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.703646    7082 generic.go:356] "Generic (PLEG): container finished" podID="def131f2-c87a-4d18-9ad3-144bfc1d7f67" containerID="198d2a831be01ae3b0504c3cca592fae36c71a817541fae58b9a6d793078c663" exitCode=0
Oct 29 22:15:55.703903 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.703820    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv" event={"ID":"def131f2-c87a-4d18-9ad3-144bfc1d7f67","Type":"ContainerDied","Data":"198d2a831be01ae3b0504c3cca592fae36c71a817541fae58b9a6d793078c663"}
Oct 29 22:15:55.707961 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.707923    7082 generic.go:356] "Generic (PLEG): container finished" podID="b1a26227-4e48-4b02-8b28-d4eaadc78c80" containerID="0760a8dcae0649dcbc31a4124f6007d9b95b5805e426c348e2908b36eacfccb5" exitCode=0
Oct 29 22:15:55.708119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.707992    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7" event={"ID":"b1a26227-4e48-4b02-8b28-d4eaadc78c80","Type":"ContainerDied","Data":"0760a8dcae0649dcbc31a4124f6007d9b95b5805e426c348e2908b36eacfccb5"}
Oct 29 22:15:55.822751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.822707    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"
Oct 29 22:15:55.826338 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.826318    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"
Oct 29 22:15:55.901916 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.901881    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def131f2-c87a-4d18-9ad3-144bfc1d7f67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def131f2-c87a-4d18-9ad3-144bfc1d7f67-plugins-config-volume\") pod \"def131f2-c87a-4d18-9ad3-144bfc1d7f67\" (UID: \"def131f2-c87a-4d18-9ad3-144bfc1d7f67\") "
Oct 29 22:15:55.902093 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.902010    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def131f2-c87a-4d18-9ad3-144bfc1d7f67-kube-api-access-jhzhj\" (UniqueName: \"kubernetes.io/projected/def131f2-c87a-4d18-9ad3-144bfc1d7f67-kube-api-access-jhzhj\") pod \"def131f2-c87a-4d18-9ad3-144bfc1d7f67\" (UID: \"def131f2-c87a-4d18-9ad3-144bfc1d7f67\") "
Oct 29 22:15:55.902093 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.902077    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a26227-4e48-4b02-8b28-d4eaadc78c80-kube-api-access-jnwp9\" (UniqueName: \"kubernetes.io/projected/b1a26227-4e48-4b02-8b28-d4eaadc78c80-kube-api-access-jnwp9\") pod \"b1a26227-4e48-4b02-8b28-d4eaadc78c80\" (UID: \"b1a26227-4e48-4b02-8b28-d4eaadc78c80\") "
Oct 29 22:15:55.902219 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.902108    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1a26227-4e48-4b02-8b28-d4eaadc78c80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a26227-4e48-4b02-8b28-d4eaadc78c80-plugins-config-volume\") pod \"b1a26227-4e48-4b02-8b28-d4eaadc78c80\" (UID: \"b1a26227-4e48-4b02-8b28-d4eaadc78c80\") "
Oct 29 22:15:55.902297 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.902263    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def131f2-c87a-4d18-9ad3-144bfc1d7f67-plugins-config-volume" pod "def131f2-c87a-4d18-9ad3-144bfc1d7f67" (UID: "def131f2-c87a-4d18-9ad3-144bfc1d7f67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:55.902601 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.902578    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def131f2-c87a-4d18-9ad3-144bfc1d7f67-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:55.902911 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.902654    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a26227-4e48-4b02-8b28-d4eaadc78c80-plugins-config-volume" pod "b1a26227-4e48-4b02-8b28-d4eaadc78c80" (UID: "b1a26227-4e48-4b02-8b28-d4eaadc78c80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:55.904617 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.904578    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a26227-4e48-4b02-8b28-d4eaadc78c80-kube-api-access-jnwp9" pod "b1a26227-4e48-4b02-8b28-d4eaadc78c80" (UID: "b1a26227-4e48-4b02-8b28-d4eaadc78c80"). InnerVolumeSpecName "kube-api-access-jnwp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:55.905667 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.905593    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def131f2-c87a-4d18-9ad3-144bfc1d7f67-kube-api-access-jhzhj" pod "def131f2-c87a-4d18-9ad3-144bfc1d7f67" (UID: "def131f2-c87a-4d18-9ad3-144bfc1d7f67"). InnerVolumeSpecName "kube-api-access-jhzhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:55.939729 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.939653    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"]
Oct 29 22:15:55.942428 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:55.942403    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a9e67f-zc5jf"]
Oct 29 22:15:56.003643 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:56.003600    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhzhj\" (UniqueName: \"kubernetes.io/projected/def131f2-c87a-4d18-9ad3-144bfc1d7f67-kube-api-access-jhzhj\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:56.003784 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:56.003658    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnwp9\" (UniqueName: \"kubernetes.io/projected/b1a26227-4e48-4b02-8b28-d4eaadc78c80-kube-api-access-jnwp9\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:56.003784 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:56.003670    7082 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a26227-4e48-4b02-8b28-d4eaadc78c80-plugins-config-volume\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:15:56.717111 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:56.717085    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"
Oct 29 22:15:56.717576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:56.717123    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv" event={"ID":"def131f2-c87a-4d18-9ad3-144bfc1d7f67","Type":"ContainerDied","Data":"04efead485f224ee14138d71c07c15fda834a115615e65cc712ef2338721dfbd"}
Oct 29 22:15:56.717576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:56.717183    7082 scope.go:122] "RemoveContainer" containerID="198d2a831be01ae3b0504c3cca592fae36c71a817541fae58b9a6d793078c663"
Oct 29 22:15:56.720669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:56.718836    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"
Oct 29 22:15:56.720669 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:56.718842    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7" event={"ID":"b1a26227-4e48-4b02-8b28-d4eaadc78c80","Type":"ContainerDied","Data":"98ceaa080c85937979dccb4e437fcdd95a51a8161458c5f36e01cb58814ff110"}
Oct 29 22:15:56.730672 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:56.730650    7082 scope.go:122] "RemoveContainer" containerID="0760a8dcae0649dcbc31a4124f6007d9b95b5805e426c348e2908b36eacfccb5"
Oct 29 22:15:56.766412 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:56.766391    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"]
Oct 29 22:15:56.777913 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:56.777893    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-6fbfd85f76-4bpxv"]
Oct 29 22:15:56.802595 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:56.802574    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"]
Oct 29 22:15:56.805673 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:56.805655    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-dcf9ddb6c-bc8f7"]
Oct 29 22:15:57.633380 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:57.633334    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a26227-4e48-4b02-8b28-d4eaadc78c80" path="/var/lib/kubelet/pods/b1a26227-4e48-4b02-8b28-d4eaadc78c80/volumes"
Oct 29 22:15:57.634052 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:57.634015    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def131f2-c87a-4d18-9ad3-144bfc1d7f67" path="/var/lib/kubelet/pods/def131f2-c87a-4d18-9ad3-144bfc1d7f67/volumes"
Oct 29 22:15:57.634642 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:15:57.634608    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe386d66-4289-4d0d-88f5-ae2bb63bc06c" path="/var/lib/kubelet/pods/fe386d66-4289-4d0d-88f5-ae2bb63bc06c/volumes"
Oct 29 22:15:58.411892 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:15:58.411856    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:16:00.979585 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:00.979544    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:16:03.588308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:03.588277    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"]
Oct 29 22:16:03.588826 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:03.588513    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy" containerID="cri-o://75b55adc30bf4aa1dd1784476c5d45c4bc3df5775c4061228befc71b0c948d29" gracePeriod=30
Oct 29 22:16:03.598396 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:03.598364    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"]
Oct 29 22:16:03.598580 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:03.598554    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" podUID="ee700b95-a323-410b-8fc8-1e5198dcf771" containerName="istio-proxy" containerID="cri-o://d6beeb2b3a39d45ff3dbc9c806f0d9e378a344b6af3931440c30135d0aff0a0b" gracePeriod=30
Oct 29 22:16:05.036445 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:05.036411    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"]
Oct 29 22:16:05.047881 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:05.047850    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"]
Oct 29 22:16:05.050439 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:05.050415    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-249a13ac-md4pp"]
Oct 29 22:16:05.052708 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:05.052687    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36d687c0-mjvbk"]
Oct 29 22:16:05.630248 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:05.630206    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2250c8f3-cb91-4604-ad64-7c675849dfd9" path="/var/lib/kubelet/pods/2250c8f3-cb91-4604-ad64-7c675849dfd9/volumes"
Oct 29 22:16:05.630715 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:05.630695    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d75350-9a77-4599-bb3f-9d833f84510e" path="/var/lib/kubelet/pods/f0d75350-9a77-4599-bb3f-9d833f84510e/volumes"
Oct 29 22:16:06.971060 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:06.971023    7082 scope.go:122] "RemoveContainer" containerID="c0a70ba57efacfd383da535aa1b018151b9b24834fb0be94e840efb154fc22e5"
Oct 29 22:16:06.981562 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:06.981540    7082 scope.go:122] "RemoveContainer" containerID="a5a23e7a6bcb79cc85f3e4ae38794e94f6ca87e72c1d84fa3cbacd58e08568ee"
Oct 29 22:16:08.846188 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:08.845930    7082 generic.go:356] "Generic (PLEG): container finished" podID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerID="75b55adc30bf4aa1dd1784476c5d45c4bc3df5775c4061228befc71b0c948d29" exitCode=0
Oct 29 22:16:08.846188 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:08.846106    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" event={"ID":"a1116dd4-7fad-45d8-8508-f0283b431cfa","Type":"ContainerDied","Data":"75b55adc30bf4aa1dd1784476c5d45c4bc3df5775c4061228befc71b0c948d29"}
Oct 29 22:16:08.848771 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:08.848745    7082 generic.go:356] "Generic (PLEG): container finished" podID="ee700b95-a323-410b-8fc8-1e5198dcf771" containerID="d6beeb2b3a39d45ff3dbc9c806f0d9e378a344b6af3931440c30135d0aff0a0b" exitCode=0
Oct 29 22:16:08.848893 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:08.848879    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" event={"ID":"ee700b95-a323-410b-8fc8-1e5198dcf771","Type":"ContainerDied","Data":"d6beeb2b3a39d45ff3dbc9c806f0d9e378a344b6af3931440c30135d0aff0a0b"}
Oct 29 22:16:08.900702 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:08.900670    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:16:09.069004 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.068969    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-kube-api-access-grzq6\" (UniqueName: \"kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-kube-api-access-grzq6\") pod \"ee700b95-a323-410b-8fc8-1e5198dcf771\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") "
Oct 29 22:16:09.069303 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.069278    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-socket\") pod \"ee700b95-a323-410b-8fc8-1e5198dcf771\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") "
Oct 29 22:16:09.069502 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.069479    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-podinfo\") pod \"ee700b95-a323-410b-8fc8-1e5198dcf771\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") "
Oct 29 22:16:09.069688 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.069671    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-envoy\") pod \"ee700b95-a323-410b-8fc8-1e5198dcf771\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") "
Oct 29 22:16:09.069859 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.069842    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-credential-socket\") pod \"ee700b95-a323-410b-8fc8-1e5198dcf771\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") "
Oct 29 22:16:09.070008 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.069992    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-token\" (UniqueName: \"kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-token\") pod \"ee700b95-a323-410b-8fc8-1e5198dcf771\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") "
Oct 29 22:16:09.070210 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.070193    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-data\") pod \"ee700b95-a323-410b-8fc8-1e5198dcf771\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") "
Oct 29 22:16:09.070351 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.070333    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee700b95-a323-410b-8fc8-1e5198dcf771-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee700b95-a323-410b-8fc8-1e5198dcf771-istiod-ca-cert\") pod \"ee700b95-a323-410b-8fc8-1e5198dcf771\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") "
Oct 29 22:16:09.070491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.070477    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-certs\") pod \"ee700b95-a323-410b-8fc8-1e5198dcf771\" (UID: \"ee700b95-a323-410b-8fc8-1e5198dcf771\") "
Oct 29 22:16:09.071319 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.071287    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-data" pod "ee700b95-a323-410b-8fc8-1e5198dcf771" (UID: "ee700b95-a323-410b-8fc8-1e5198dcf771"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.071479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.071309    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee700b95-a323-410b-8fc8-1e5198dcf771-istiod-ca-cert" pod "ee700b95-a323-410b-8fc8-1e5198dcf771" (UID: "ee700b95-a323-410b-8fc8-1e5198dcf771"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:09.071544 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.071473    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-credential-socket" pod "ee700b95-a323-410b-8fc8-1e5198dcf771" (UID: "ee700b95-a323-410b-8fc8-1e5198dcf771"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.071651 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.071611    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-certs" pod "ee700b95-a323-410b-8fc8-1e5198dcf771" (UID: "ee700b95-a323-410b-8fc8-1e5198dcf771"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.071765 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.071687    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-socket" pod "ee700b95-a323-410b-8fc8-1e5198dcf771" (UID: "ee700b95-a323-410b-8fc8-1e5198dcf771"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.076109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.075406    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-envoy" pod "ee700b95-a323-410b-8fc8-1e5198dcf771" (UID: "ee700b95-a323-410b-8fc8-1e5198dcf771"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.076109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.075868    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-token" pod "ee700b95-a323-410b-8fc8-1e5198dcf771" (UID: "ee700b95-a323-410b-8fc8-1e5198dcf771"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:09.076109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.076043    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-podinfo" pod "ee700b95-a323-410b-8fc8-1e5198dcf771" (UID: "ee700b95-a323-410b-8fc8-1e5198dcf771"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:09.076109 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.076073    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-kube-api-access-grzq6" pod "ee700b95-a323-410b-8fc8-1e5198dcf771" (UID: "ee700b95-a323-410b-8fc8-1e5198dcf771"). InnerVolumeSpecName "kube-api-access-grzq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:09.077752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.077728    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"]
Oct 29 22:16:09.172465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.172250    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.172465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.172288    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.172465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.172302    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.172465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.172317    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.172465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.172330    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.172465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.172342    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee700b95-a323-410b-8fc8-1e5198dcf771-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.172465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.172353    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.172465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.172366    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grzq6\" (UniqueName: \"kubernetes.io/projected/ee700b95-a323-410b-8fc8-1e5198dcf771-kube-api-access-grzq6\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.172465 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.172379    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee700b95-a323-410b-8fc8-1e5198dcf771-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.839308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.839276    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:16:09.865268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.865228    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z" event={"ID":"ee700b95-a323-410b-8fc8-1e5198dcf771","Type":"ContainerDied","Data":"4e1d11dbd0048728c7e7388c549899b8e2753a8ceb041e94d2cd53cc99ed3138"}
Oct 29 22:16:09.866697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.865856    7082 scope.go:122] "RemoveContainer" containerID="d6beeb2b3a39d45ff3dbc9c806f0d9e378a344b6af3931440c30135d0aff0a0b"
Oct 29 22:16:09.866697 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.866034    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"
Oct 29 22:16:09.872969 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.872901    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f" event={"ID":"a1116dd4-7fad-45d8-8508-f0283b431cfa","Type":"ContainerDied","Data":"4605b5f36e43a05bbcaf81eb7519bbdb984c8a2c310fb390c7d5f8dd5e49f883"}
Oct 29 22:16:09.873094 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.873009    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"
Oct 29 22:16:09.883070 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.883051    7082 scope.go:122] "RemoveContainer" containerID="75b55adc30bf4aa1dd1784476c5d45c4bc3df5775c4061228befc71b0c948d29"
Oct 29 22:16:09.945289 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.924614    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-token\" (UniqueName: \"kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-token\") pod \"a1116dd4-7fad-45d8-8508-f0283b431cfa\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") "
Oct 29 22:16:09.945489 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.945381    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-envoy\") pod \"a1116dd4-7fad-45d8-8508-f0283b431cfa\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") "
Oct 29 22:16:09.945489 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.945412    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-podinfo\") pod \"a1116dd4-7fad-45d8-8508-f0283b431cfa\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") "
Oct 29 22:16:09.945613 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.945486    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-kube-api-access-lp6pb\" (UniqueName: \"kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-kube-api-access-lp6pb\") pod \"a1116dd4-7fad-45d8-8508-f0283b431cfa\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") "
Oct 29 22:16:09.945613 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.945565    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-socket\") pod \"a1116dd4-7fad-45d8-8508-f0283b431cfa\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") "
Oct 29 22:16:09.945800 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.945612    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-certs\") pod \"a1116dd4-7fad-45d8-8508-f0283b431cfa\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") "
Oct 29 22:16:09.945800 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.945681    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1116dd4-7fad-45d8-8508-f0283b431cfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1116dd4-7fad-45d8-8508-f0283b431cfa-istiod-ca-cert\") pod \"a1116dd4-7fad-45d8-8508-f0283b431cfa\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") "
Oct 29 22:16:09.945800 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.945715    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-data\") pod \"a1116dd4-7fad-45d8-8508-f0283b431cfa\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") "
Oct 29 22:16:09.945800 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.945780    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-credential-socket\") pod \"a1116dd4-7fad-45d8-8508-f0283b431cfa\" (UID: \"a1116dd4-7fad-45d8-8508-f0283b431cfa\") "
Oct 29 22:16:09.946211 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.946184    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-token" pod "a1116dd4-7fad-45d8-8508-f0283b431cfa" (UID: "a1116dd4-7fad-45d8-8508-f0283b431cfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:09.946376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.946329    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-socket" pod "a1116dd4-7fad-45d8-8508-f0283b431cfa" (UID: "a1116dd4-7fad-45d8-8508-f0283b431cfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.946496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.946470    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-credential-socket" pod "a1116dd4-7fad-45d8-8508-f0283b431cfa" (UID: "a1116dd4-7fad-45d8-8508-f0283b431cfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.946718 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.946695    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-data" pod "a1116dd4-7fad-45d8-8508-f0283b431cfa" (UID: "a1116dd4-7fad-45d8-8508-f0283b431cfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.946868 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.946848    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1116dd4-7fad-45d8-8508-f0283b431cfa-istiod-ca-cert" pod "a1116dd4-7fad-45d8-8508-f0283b431cfa" (UID: "a1116dd4-7fad-45d8-8508-f0283b431cfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:09.947077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.947059    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-certs" pod "a1116dd4-7fad-45d8-8508-f0283b431cfa" (UID: "a1116dd4-7fad-45d8-8508-f0283b431cfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.949821 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.949794    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-kube-api-access-lp6pb" pod "a1116dd4-7fad-45d8-8508-f0283b431cfa" (UID: "a1116dd4-7fad-45d8-8508-f0283b431cfa"). InnerVolumeSpecName "kube-api-access-lp6pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:09.949922 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.949887    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-envoy" pod "a1116dd4-7fad-45d8-8508-f0283b431cfa" (UID: "a1116dd4-7fad-45d8-8508-f0283b431cfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.950588 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.950314    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-podinfo" pod "a1116dd4-7fad-45d8-8508-f0283b431cfa" (UID: "a1116dd4-7fad-45d8-8508-f0283b431cfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:09.985282 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.985255    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"]
Oct 29 22:16:09.989474 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:09.989437    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-9lt4z"]
Oct 29 22:16:10.047773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:10.047700    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:10.047773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:10.047731    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:10.047773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:10.047745    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:10.047773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:10.047758    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp6pb\" (UniqueName: \"kubernetes.io/projected/a1116dd4-7fad-45d8-8508-f0283b431cfa-kube-api-access-lp6pb\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:10.047773 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:10.047772    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:10.048064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:10.047785    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:10.048064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:10.047799    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1116dd4-7fad-45d8-8508-f0283b431cfa-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:10.048064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:10.047811    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:10.048064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:10.047825    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1116dd4-7fad-45d8-8508-f0283b431cfa-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:10.219878 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:10.219844    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"]
Oct 29 22:16:10.222728 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:10.222702    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-l2w9f"]
Oct 29 22:16:11.630512 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:11.630472    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" path="/var/lib/kubelet/pods/a1116dd4-7fad-45d8-8508-f0283b431cfa/volumes"
Oct 29 22:16:11.631107 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:11.631089    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee700b95-a323-410b-8fc8-1e5198dcf771" path="/var/lib/kubelet/pods/ee700b95-a323-410b-8fc8-1e5198dcf771/volumes"
Oct 29 22:16:12.043031 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:12.042950    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"]
Oct 29 22:16:12.054906 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:12.054878    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"]
Oct 29 22:16:12.056971 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:12.056957    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f87b5d-hdvc4"]
Oct 29 22:16:12.058836 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:12.058815    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bac048a8-mkkrm"]
Oct 29 22:16:13.632962 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:13.632925    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b319e77b-08ea-4820-99f9-98859acbbcbb" path="/var/lib/kubelet/pods/b319e77b-08ea-4820-99f9-98859acbbcbb/volumes"
Oct 29 22:16:13.633496 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:13.633478    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd3b72f8-23bb-4617-a074-384c32e35667" path="/var/lib/kubelet/pods/cd3b72f8-23bb-4617-a074-384c32e35667/volumes"
Oct 29 22:16:17.219898 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.219865    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"]
Oct 29 22:16:17.220325 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220310    7082 cpu_manager.go:414] "containerMap: removing container" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy"
Oct 29 22:16:17.220376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220326    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy"
Oct 29 22:16:17.220376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220343    7082 cpu_manager.go:414] "containerMap: removing container" podUID="fe386d66-4289-4d0d-88f5-ae2bb63bc06c" containerName="revoke-keys"
Oct 29 22:16:17.220376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220350    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe386d66-4289-4d0d-88f5-ae2bb63bc06c" containerName="revoke-keys"
Oct 29 22:16:17.220376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220361    7082 cpu_manager.go:414] "containerMap: removing container" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy"
Oct 29 22:16:17.220376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220366    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy"
Oct 29 22:16:17.220376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220373    7082 cpu_manager.go:414] "containerMap: removing container" podUID="ee700b95-a323-410b-8fc8-1e5198dcf771" containerName="istio-proxy"
Oct 29 22:16:17.220376 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220378    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee700b95-a323-410b-8fc8-1e5198dcf771" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220389    7082 cpu_manager.go:414] "containerMap: removing container" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220393    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220402    7082 cpu_manager.go:414] "containerMap: removing container" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220407    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220413    7082 cpu_manager.go:414] "containerMap: removing container" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220417    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220426    7082 cpu_manager.go:414] "containerMap: removing container" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220431    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220441    7082 cpu_manager.go:414] "containerMap: removing container" podUID="b1a26227-4e48-4b02-8b28-d4eaadc78c80" containerName="payload-pre-processing"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220446    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a26227-4e48-4b02-8b28-d4eaadc78c80" containerName="payload-pre-processing"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220454    7082 cpu_manager.go:414] "containerMap: removing container" podUID="def131f2-c87a-4d18-9ad3-144bfc1d7f67" containerName="payload-processing"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220459    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def131f2-c87a-4d18-9ad3-144bfc1d7f67" containerName="payload-processing"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220467    7082 cpu_manager.go:414] "containerMap: removing container" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220472    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220480    7082 cpu_manager.go:414] "containerMap: removing container" podUID="f7380f6d-cac7-48e0-b312-41612a211252" containerName="cleanup"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220484    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7380f6d-cac7-48e0-b312-41612a211252" containerName="cleanup"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220492    7082 cpu_manager.go:414] "containerMap: removing container" podUID="bf70914c-77a1-427d-8c85-903954bde9dd" containerName="cleanup"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220496    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf70914c-77a1-427d-8c85-903954bde9dd" containerName="cleanup"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220561    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe386d66-4289-4d0d-88f5-ae2bb63bc06c" containerName="revoke-keys"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220570    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220576    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee700b95-a323-410b-8fc8-1e5198dcf771" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220583    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7fa2cd-1234-4b80-b869-3ab7019d1cea" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220590    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1116dd4-7fad-45d8-8508-f0283b431cfa" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220596    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e69aa18-c109-4098-b08a-67695f24fa34" containerName="istio-proxy"
Oct 29 22:16:17.220604 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220612    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2febe4-d73f-42e3-bf74-ce53ba139b78" containerName="istio-proxy"
Oct 29 22:16:17.221385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220620    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a26227-4e48-4b02-8b28-d4eaadc78c80" containerName="payload-pre-processing"
Oct 29 22:16:17.221385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220641    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="def131f2-c87a-4d18-9ad3-144bfc1d7f67" containerName="payload-processing"
Oct 29 22:16:17.221385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220647    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3833557-dcfc-4ed5-8415-cca71bba35e7" containerName="istio-proxy"
Oct 29 22:16:17.221385 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.220653    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="4301312f-6325-4e8c-9df7-8e4153fbcea9" containerName="istio-proxy"
Oct 29 22:16:17.225499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.225483    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"
Oct 29 22:16:17.246115 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.246081    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"]
Oct 29 22:16:17.330272 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.330241    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:16:17.330447 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.330296    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:16:17.332933 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.332911    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:16:17.333032 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.332937    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:16:17.358462 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.358439    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433477a8-5832-4475-989c-e01be3c42e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c639c5-x5w72\" (UID: \"433477a8-5832-4475-989c-e01be3c42e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"
Oct 29 22:16:17.358569 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.358554    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t55fp\" (UniqueName: \"kubernetes.io/projected/433477a8-5832-4475-989c-e01be3c42e88-kube-api-access-t55fp\") pod \"maas-api-revoke-sub-28c639c5-x5w72\" (UID: \"433477a8-5832-4475-989c-e01be3c42e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"
Oct 29 22:16:17.358650 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.358611    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433477a8-5832-4475-989c-e01be3c42e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c639c5-x5w72\" (UID: \"433477a8-5832-4475-989c-e01be3c42e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"
Oct 29 22:16:17.459537 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.459503    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t55fp\" (UniqueName: \"kubernetes.io/projected/433477a8-5832-4475-989c-e01be3c42e88-kube-api-access-t55fp\") pod \"maas-api-revoke-sub-28c639c5-x5w72\" (UID: \"433477a8-5832-4475-989c-e01be3c42e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"
Oct 29 22:16:17.459703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.459575    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433477a8-5832-4475-989c-e01be3c42e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c639c5-x5w72\" (UID: \"433477a8-5832-4475-989c-e01be3c42e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"
Oct 29 22:16:17.459703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.459690    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433477a8-5832-4475-989c-e01be3c42e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c639c5-x5w72\" (UID: \"433477a8-5832-4475-989c-e01be3c42e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"
Oct 29 22:16:17.460308 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.460284    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433477a8-5832-4475-989c-e01be3c42e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28c639c5-x5w72\" (UID: \"433477a8-5832-4475-989c-e01be3c42e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"
Oct 29 22:16:17.462164 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.462063    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433477a8-5832-4475-989c-e01be3c42e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28c639c5-x5w72\" (UID: \"433477a8-5832-4475-989c-e01be3c42e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"
Oct 29 22:16:17.482364 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.482303    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t55fp\" (UniqueName: \"kubernetes.io/projected/433477a8-5832-4475-989c-e01be3c42e88-kube-api-access-t55fp\") pod \"maas-api-revoke-sub-28c639c5-x5w72\" (UID: \"433477a8-5832-4475-989c-e01be3c42e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"
Oct 29 22:16:17.541979 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.541936    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"
Oct 29 22:16:17.850256 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.850227    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"]
Oct 29 22:16:17.852472 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:16:17.852444    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433477a8_5832_4475_989c_e01be3c42e88.slice/crio-c83a1b121f02888b50b78f27905075d221d1e496c2d286150c74b815cf31b1ba WatchSource:0}: Error finding container c83a1b121f02888b50b78f27905075d221d1e496c2d286150c74b815cf31b1ba: Status 404 returned error can't find the container with id c83a1b121f02888b50b78f27905075d221d1e496c2d286150c74b815cf31b1ba
Oct 29 22:16:17.959375 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:17.959341    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72" event={"ID":"433477a8-5832-4475-989c-e01be3c42e88","Type":"ContainerStarted","Data":"c83a1b121f02888b50b78f27905075d221d1e496c2d286150c74b815cf31b1ba"}
Oct 29 22:16:18.967958 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:18.967925    7082 generic.go:356] "Generic (PLEG): container finished" podID="433477a8-5832-4475-989c-e01be3c42e88" containerID="db66cc966a51729220e3dc6fcc8465de48970ca334c9ff88ccf59a105d9271dc" exitCode=0
Oct 29 22:16:18.968372 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:18.968014    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72" event={"ID":"433477a8-5832-4475-989c-e01be3c42e88","Type":"ContainerDied","Data":"db66cc966a51729220e3dc6fcc8465de48970ca334c9ff88ccf59a105d9271dc"}
Oct 29 22:16:20.126291 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:20.126266    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"
Oct 29 22:16:20.287408 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:20.287310    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433477a8-5832-4475-989c-e01be3c42e88-kube-api-access-t55fp\" (UniqueName: \"kubernetes.io/projected/433477a8-5832-4475-989c-e01be3c42e88-kube-api-access-t55fp\") pod \"433477a8-5832-4475-989c-e01be3c42e88\" (UID: \"433477a8-5832-4475-989c-e01be3c42e88\") "
Oct 29 22:16:20.287600 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:20.287419    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/433477a8-5832-4475-989c-e01be3c42e88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433477a8-5832-4475-989c-e01be3c42e88-maas-api-xks-ca\") pod \"433477a8-5832-4475-989c-e01be3c42e88\" (UID: \"433477a8-5832-4475-989c-e01be3c42e88\") "
Oct 29 22:16:20.287600 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:20.287458    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/433477a8-5832-4475-989c-e01be3c42e88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433477a8-5832-4475-989c-e01be3c42e88-maas-api-openshift-ca\") pod \"433477a8-5832-4475-989c-e01be3c42e88\" (UID: \"433477a8-5832-4475-989c-e01be3c42e88\") "
Oct 29 22:16:20.287946 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:20.287915    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433477a8-5832-4475-989c-e01be3c42e88-maas-api-openshift-ca" pod "433477a8-5832-4475-989c-e01be3c42e88" (UID: "433477a8-5832-4475-989c-e01be3c42e88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:20.289690 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:20.289660    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/433477a8-5832-4475-989c-e01be3c42e88-maas-api-xks-ca" pod "433477a8-5832-4475-989c-e01be3c42e88" (UID: "433477a8-5832-4475-989c-e01be3c42e88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:16:20.289790 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:20.289733    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433477a8-5832-4475-989c-e01be3c42e88-kube-api-access-t55fp" pod "433477a8-5832-4475-989c-e01be3c42e88" (UID: "433477a8-5832-4475-989c-e01be3c42e88"). InnerVolumeSpecName "kube-api-access-t55fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:20.389077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:20.389038    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t55fp\" (UniqueName: \"kubernetes.io/projected/433477a8-5832-4475-989c-e01be3c42e88-kube-api-access-t55fp\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:20.389077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:20.389071    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/433477a8-5832-4475-989c-e01be3c42e88-maas-api-xks-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:20.389077 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:20.389080    7082 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/433477a8-5832-4475-989c-e01be3c42e88-maas-api-openshift-ca\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:20.984009 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:20.983974    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72" event={"ID":"433477a8-5832-4475-989c-e01be3c42e88","Type":"ContainerDied","Data":"c83a1b121f02888b50b78f27905075d221d1e496c2d286150c74b815cf31b1ba"}
Oct 29 22:16:20.984009 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:20.984003    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28c639c5-x5w72"
Oct 29 22:16:20.984253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:20.984029    7082 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c83a1b121f02888b50b78f27905075d221d1e496c2d286150c74b815cf31b1ba"
Oct 29 22:16:24.274369 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:24.274317    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" containerID="cri-o://357d2a0fc2fdb8071a7f13d320d9385b14577655eab7ee9060441f1ab8e38440" gracePeriod=45
Oct 29 22:16:24.635011 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:24.634968    7082 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": dial tcp 10.134.0.100:8000: connect: connection refused"
Oct 29 22:16:25.016157 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.015794    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:16:25.041245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.040763    7082 generic.go:356] "Generic (PLEG): container finished" podID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerID="357d2a0fc2fdb8071a7f13d320d9385b14577655eab7ee9060441f1ab8e38440" exitCode=0
Oct 29 22:16:25.041245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.040868    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" event={"ID":"55aac7b4-d385-4ba9-b5a0-7528b481f142","Type":"ContainerDied","Data":"357d2a0fc2fdb8071a7f13d320d9385b14577655eab7ee9060441f1ab8e38440"}
Oct 29 22:16:25.041245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.040910    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t" event={"ID":"55aac7b4-d385-4ba9-b5a0-7528b481f142","Type":"ContainerDied","Data":"8731253ca7a95d5e381b21e5db9813f00821338bf0cfee0208ad374a9122ae2d"}
Oct 29 22:16:25.041245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.040936    7082 scope.go:122] "RemoveContainer" containerID="357d2a0fc2fdb8071a7f13d320d9385b14577655eab7ee9060441f1ab8e38440"
Oct 29 22:16:25.041245 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.041114    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"
Oct 29 22:16:25.072822 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.071801    7082 scope.go:122] "RemoveContainer" containerID="357d2a0fc2fdb8071a7f13d320d9385b14577655eab7ee9060441f1ab8e38440"
Oct 29 22:16:25.091678 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:16:25.088165    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357d2a0fc2fdb8071a7f13d320d9385b14577655eab7ee9060441f1ab8e38440\": container with ID starting with 357d2a0fc2fdb8071a7f13d320d9385b14577655eab7ee9060441f1ab8e38440 not found: ID does not exist" containerID="357d2a0fc2fdb8071a7f13d320d9385b14577655eab7ee9060441f1ab8e38440"
Oct 29 22:16:25.091678 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.088219    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357d2a0fc2fdb8071a7f13d320d9385b14577655eab7ee9060441f1ab8e38440"} err="failed to get container status \"357d2a0fc2fdb8071a7f13d320d9385b14577655eab7ee9060441f1ab8e38440\": rpc error: code = NotFound desc = could not find container \"357d2a0fc2fdb8071a7f13d320d9385b14577655eab7ee9060441f1ab8e38440\": container with ID starting with 357d2a0fc2fdb8071a7f13d320d9385b14577655eab7ee9060441f1ab8e38440 not found: ID does not exist"
Oct 29 22:16:25.135237 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.132476    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-model-cache\") pod \"55aac7b4-d385-4ba9-b5a0-7528b481f142\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") "
Oct 29 22:16:25.135237 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.132544    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55aac7b4-d385-4ba9-b5a0-7528b481f142-kube-api-access-s6fbl\" (UniqueName: \"kubernetes.io/projected/55aac7b4-d385-4ba9-b5a0-7528b481f142-kube-api-access-s6fbl\") pod \"55aac7b4-d385-4ba9-b5a0-7528b481f142\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") "
Oct 29 22:16:25.135237 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.132578    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-dshm\") pod \"55aac7b4-d385-4ba9-b5a0-7528b481f142\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") "
Oct 29 22:16:25.135237 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.132619    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-tmp-dir\") pod \"55aac7b4-d385-4ba9-b5a0-7528b481f142\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") "
Oct 29 22:16:25.135237 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.132833    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-home\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-home\") pod \"55aac7b4-d385-4ba9-b5a0-7528b481f142\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") "
Oct 29 22:16:25.135237 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.132924    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55aac7b4-d385-4ba9-b5a0-7528b481f142-tls-certs\" (UniqueName: \"kubernetes.io/secret/55aac7b4-d385-4ba9-b5a0-7528b481f142-tls-certs\") pod \"55aac7b4-d385-4ba9-b5a0-7528b481f142\" (UID: \"55aac7b4-d385-4ba9-b5a0-7528b481f142\") "
Oct 29 22:16:25.135237 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.134657    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-tmp-dir" pod "55aac7b4-d385-4ba9-b5a0-7528b481f142" (UID: "55aac7b4-d385-4ba9-b5a0-7528b481f142"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:25.135237 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.134916    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-model-cache" pod "55aac7b4-d385-4ba9-b5a0-7528b481f142" (UID: "55aac7b4-d385-4ba9-b5a0-7528b481f142"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:25.136205 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.136154    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-home" pod "55aac7b4-d385-4ba9-b5a0-7528b481f142" (UID: "55aac7b4-d385-4ba9-b5a0-7528b481f142"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:25.145986 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.145957    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-dshm" pod "55aac7b4-d385-4ba9-b5a0-7528b481f142" (UID: "55aac7b4-d385-4ba9-b5a0-7528b481f142"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:25.146181 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.146087    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55aac7b4-d385-4ba9-b5a0-7528b481f142-tls-certs" pod "55aac7b4-d385-4ba9-b5a0-7528b481f142" (UID: "55aac7b4-d385-4ba9-b5a0-7528b481f142"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:16:25.151403 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.151329    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55aac7b4-d385-4ba9-b5a0-7528b481f142-kube-api-access-s6fbl" pod "55aac7b4-d385-4ba9-b5a0-7528b481f142" (UID: "55aac7b4-d385-4ba9-b5a0-7528b481f142"). InnerVolumeSpecName "kube-api-access-s6fbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:25.234446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.234197    7082 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-model-cache\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.234446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.234245    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6fbl\" (UniqueName: \"kubernetes.io/projected/55aac7b4-d385-4ba9-b5a0-7528b481f142-kube-api-access-s6fbl\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.234446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.234260    7082 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-dshm\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.234446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.234273    7082 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-tmp-dir\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.234446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.234284    7082 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55aac7b4-d385-4ba9-b5a0-7528b481f142-home\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.234446 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.234296    7082 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55aac7b4-d385-4ba9-b5a0-7528b481f142-tls-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.386243 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.386208    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"]
Oct 29 22:16:25.397857 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.397821    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/distinct-2-w3-8a9e5f-kserve-688ff84c9f-66z5t"]
Oct 29 22:16:25.633192 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:25.633160    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" path="/var/lib/kubelet/pods/55aac7b4-d385-4ba9-b5a0-7528b481f142/volumes"
Oct 29 22:16:28.411787 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:16:28.411746    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:16:29.480100 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:29.480053    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:16:35.083705 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:35.081677    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-f7d6b5c7c-7sl7d"]
Oct 29 22:16:40.116130 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:40.116088    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"]
Oct 29 22:16:40.116701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:40.116350    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerName="istio-proxy" containerID="cri-o://2faa7e03ec44d32988b2b89c3b88e65a35e3d02a1926e28262215b309747cb4b" gracePeriod=30
Oct 29 22:16:40.132309 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:40.132262    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"]
Oct 29 22:16:40.132563 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:40.132535    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy" containerID="cri-o://bccbd6fa084f2a6ee5713128fc2175c95a417810cbcba8583c24b8cce74c358e" gracePeriod=30
Oct 29 22:16:40.135451 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:40.135424    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"]
Oct 29 22:16:40.135984 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:40.135951    7082 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy" containerID="cri-o://4608140203d245d18679ad7e6432b15765ed1c10bb2a4a0fc92d01aca22ae25c" gracePeriod=30
Oct 29 22:16:41.291302 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:41.291269    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8468f79957-dxp8f_fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c/authorino/0.log"
Oct 29 22:16:41.774140 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:41.774109    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_developer-portal-controller-66d8dbb8c8-b2ll6_ba1d0015-3c5e-450d-a435-d33ab885f265/manager/0.log"
Oct 29 22:16:42.787723 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:42.787692    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6b575997cc-zthg8_b8178c0e-7643-499f-9dc0-e04329809348/manager/1.log"
Oct 29 22:16:43.005994 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:43.005945    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-f7d6b5c7c-7sl7d_72d022e0-2c45-448f-8bfb-77d5d75d6f8f/limitador/0.log"
Oct 29 22:16:43.846889 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:43.846859    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6b575997cc-zthg8_b8178c0e-7643-499f-9dc0-e04329809348/manager/0.log"
Oct 29 22:16:44.823862 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:44.823822    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9cw8s_522ce2f8-9d09-478b-8575-b2dabf74a61e/manager/0.log"
Oct 29 22:16:45.390502 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.390472    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 22:16:45.393848 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.393768    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-data\") pod \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") "
Oct 29 22:16:45.393848 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.393823    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-kube-api-access-qgx49\" (UniqueName: \"kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-kube-api-access-qgx49\") pod \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") "
Oct 29 22:16:45.394064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.393896    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-socket\") pod \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") "
Oct 29 22:16:45.394064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.393932    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-token\" (UniqueName: \"kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-token\") pod \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") "
Oct 29 22:16:45.394064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.393976    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-podinfo\") pod \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") "
Oct 29 22:16:45.394064 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.394033    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-certs\") pod \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") "
Oct 29 22:16:45.394524 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.394085    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-credential-socket\") pod \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") "
Oct 29 22:16:45.394524 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.394154    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-envoy\") pod \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") "
Oct 29 22:16:45.394524 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.394181    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istiod-ca-cert\") pod \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\" (UID: \"9c6e140c-ea25-446f-a3b2-42d49e6634e5\") "
Oct 29 22:16:45.394524 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.394482    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-credential-socket" pod "9c6e140c-ea25-446f-a3b2-42d49e6634e5" (UID: "9c6e140c-ea25-446f-a3b2-42d49e6634e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.394524 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.394471    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-certs" pod "9c6e140c-ea25-446f-a3b2-42d49e6634e5" (UID: "9c6e140c-ea25-446f-a3b2-42d49e6634e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.395491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.394643    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.395491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.394663    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.395491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.394843    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istiod-ca-cert" pod "9c6e140c-ea25-446f-a3b2-42d49e6634e5" (UID: "9c6e140c-ea25-446f-a3b2-42d49e6634e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:45.395491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.395083    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-socket" pod "9c6e140c-ea25-446f-a3b2-42d49e6634e5" (UID: "9c6e140c-ea25-446f-a3b2-42d49e6634e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.395491 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.395184    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-data" pod "9c6e140c-ea25-446f-a3b2-42d49e6634e5" (UID: "9c6e140c-ea25-446f-a3b2-42d49e6634e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.398061 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.398026    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-envoy" pod "9c6e140c-ea25-446f-a3b2-42d49e6634e5" (UID: "9c6e140c-ea25-446f-a3b2-42d49e6634e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.398169 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.398069    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-podinfo" pod "9c6e140c-ea25-446f-a3b2-42d49e6634e5" (UID: "9c6e140c-ea25-446f-a3b2-42d49e6634e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:45.401384 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.401364    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-token" pod "9c6e140c-ea25-446f-a3b2-42d49e6634e5" (UID: "9c6e140c-ea25-446f-a3b2-42d49e6634e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.401885 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.401862    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-kube-api-access-qgx49" pod "9c6e140c-ea25-446f-a3b2-42d49e6634e5" (UID: "9c6e140c-ea25-446f-a3b2-42d49e6634e5"). InnerVolumeSpecName "kube-api-access-qgx49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.422467 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.422436    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 22:16:45.456644 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.456599    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 22:16:45.497214 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497177    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-credential-socket\") pod \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") "
Oct 29 22:16:45.497479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497231    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-podinfo\") pod \"f85ac047-ae41-4a6d-9711-87e13cf1917b\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") "
Oct 29 22:16:45.497479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497265    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-kube-api-access-cf488\" (UniqueName: \"kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-kube-api-access-cf488\") pod \"f85ac047-ae41-4a6d-9711-87e13cf1917b\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") "
Oct 29 22:16:45.497479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497295    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-credential-socket\") pod \"f85ac047-ae41-4a6d-9711-87e13cf1917b\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") "
Oct 29 22:16:45.497479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497353    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-kube-api-access-l97jv\" (UniqueName: \"kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-kube-api-access-l97jv\") pod \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") "
Oct 29 22:16:45.497479 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497453    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-data\") pod \"f85ac047-ae41-4a6d-9711-87e13cf1917b\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") "
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497498    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-certs\") pod \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") "
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497544    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85ac047-ae41-4a6d-9711-87e13cf1917b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85ac047-ae41-4a6d-9711-87e13cf1917b-istiod-ca-cert\") pod \"f85ac047-ae41-4a6d-9711-87e13cf1917b\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") "
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497545    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-credential-socket" pod "65dfc02a-c81d-4366-a1a5-a5808d335aaf" (UID: "65dfc02a-c81d-4366-a1a5-a5808d335aaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497573    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-envoy\") pod \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") "
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497601    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-data\") pod \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") "
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497657    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-socket\") pod \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") "
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497714    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-podinfo\") pod \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") "
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497747    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-envoy\") pod \"f85ac047-ae41-4a6d-9711-87e13cf1917b\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") "
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497779    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istiod-ca-cert\") pod \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") "
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497806    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-socket\") pod \"f85ac047-ae41-4a6d-9711-87e13cf1917b\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") "
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.497853    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-token\" (UniqueName: \"kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-token\") pod \"f85ac047-ae41-4a6d-9711-87e13cf1917b\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") "
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498057    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85ac047-ae41-4a6d-9711-87e13cf1917b-istiod-ca-cert" pod "f85ac047-ae41-4a6d-9711-87e13cf1917b" (UID: "f85ac047-ae41-4a6d-9711-87e13cf1917b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498314    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-credential-socket" pod "f85ac047-ae41-4a6d-9711-87e13cf1917b" (UID: "f85ac047-ae41-4a6d-9711-87e13cf1917b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498335    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-data" pod "f85ac047-ae41-4a6d-9711-87e13cf1917b" (UID: "f85ac047-ae41-4a6d-9711-87e13cf1917b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498359    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498377    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498391    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498404    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85ac047-ae41-4a6d-9711-87e13cf1917b-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498418    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498430    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498443    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498457    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6e140c-ea25-446f-a3b2-42d49e6634e5-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498471    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgx49\" (UniqueName: \"kubernetes.io/projected/9c6e140c-ea25-446f-a3b2-42d49e6634e5-kube-api-access-qgx49\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498727    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-socket" pod "f85ac047-ae41-4a6d-9711-87e13cf1917b" (UID: "f85ac047-ae41-4a6d-9711-87e13cf1917b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.498776    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istiod-ca-cert" pod "65dfc02a-c81d-4366-a1a5-a5808d335aaf" (UID: "65dfc02a-c81d-4366-a1a5-a5808d335aaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.499071    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-certs" pod "65dfc02a-c81d-4366-a1a5-a5808d335aaf" (UID: "65dfc02a-c81d-4366-a1a5-a5808d335aaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.499369    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-data" pod "65dfc02a-c81d-4366-a1a5-a5808d335aaf" (UID: "65dfc02a-c81d-4366-a1a5-a5808d335aaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.500415 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.499950    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-socket" pod "65dfc02a-c81d-4366-a1a5-a5808d335aaf" (UID: "65dfc02a-c81d-4366-a1a5-a5808d335aaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.502015 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.500893    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-kube-api-access-cf488" pod "f85ac047-ae41-4a6d-9711-87e13cf1917b" (UID: "f85ac047-ae41-4a6d-9711-87e13cf1917b"). InnerVolumeSpecName "kube-api-access-cf488". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.502015 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.501718    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-podinfo" pod "65dfc02a-c81d-4366-a1a5-a5808d335aaf" (UID: "65dfc02a-c81d-4366-a1a5-a5808d335aaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:45.503053 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.503025    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-token" pod "f85ac047-ae41-4a6d-9711-87e13cf1917b" (UID: "f85ac047-ae41-4a6d-9711-87e13cf1917b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.503452 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.503434    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-kube-api-access-l97jv" pod "65dfc02a-c81d-4366-a1a5-a5808d335aaf" (UID: "65dfc02a-c81d-4366-a1a5-a5808d335aaf"). InnerVolumeSpecName "kube-api-access-l97jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.503599 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.503577    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-envoy" pod "65dfc02a-c81d-4366-a1a5-a5808d335aaf" (UID: "65dfc02a-c81d-4366-a1a5-a5808d335aaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.504938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.504917    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-envoy" pod "f85ac047-ae41-4a6d-9711-87e13cf1917b" (UID: "f85ac047-ae41-4a6d-9711-87e13cf1917b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.505752 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.505727    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-podinfo" pod "f85ac047-ae41-4a6d-9711-87e13cf1917b" (UID: "f85ac047-ae41-4a6d-9711-87e13cf1917b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:45.599402 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599371    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-token\" (UniqueName: \"kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-token\") pod \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\" (UID: \"65dfc02a-c81d-4366-a1a5-a5808d335aaf\") "
Oct 29 22:16:45.599576 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599413    7082 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-certs\") pod \"f85ac047-ae41-4a6d-9711-87e13cf1917b\" (UID: \"f85ac047-ae41-4a6d-9711-87e13cf1917b\") "
Oct 29 22:16:45.599736 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599720    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf488\" (UniqueName: \"kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-kube-api-access-cf488\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.599811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599736    7082 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-credential-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.599811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599746    7082 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l97jv\" (UniqueName: \"kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-kube-api-access-l97jv\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.599811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599755    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.599811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599764    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.599811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599772    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.599811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599779    7082 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-data\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.599811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599786    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65dfc02a-c81d-4366-a1a5-a5808d335aaf-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.599811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599793    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.599811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599801    7082 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-envoy\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.599811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599808    7082 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istiod-ca-cert\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.599811 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599816    7082 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-socket\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.600254 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599823    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.600254 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599832    7082 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85ac047-ae41-4a6d-9711-87e13cf1917b-istio-podinfo\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.600254 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.599830    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-certs" pod "f85ac047-ae41-4a6d-9711-87e13cf1917b" (UID: "f85ac047-ae41-4a6d-9711-87e13cf1917b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.602031 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.602002    7082 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-token" pod "65dfc02a-c81d-4366-a1a5-a5808d335aaf" (UID: "65dfc02a-c81d-4366-a1a5-a5808d335aaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.700911 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.700866    7082 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65dfc02a-c81d-4366-a1a5-a5808d335aaf-istio-token\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.700911 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:45.700900    7082 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85ac047-ae41-4a6d-9711-87e13cf1917b-workload-certs\") on node \"ip-10-0-136-38.ec2.internal\" DevicePath \"\""
Oct 29 22:16:46.195204 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.195162    7082 generic.go:356] "Generic (PLEG): container finished" podID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerID="4608140203d245d18679ad7e6432b15765ed1c10bb2a4a0fc92d01aca22ae25c" exitCode=0
Oct 29 22:16:46.195405 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.195223    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"
Oct 29 22:16:46.195405 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.195243    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" event={"ID":"f85ac047-ae41-4a6d-9711-87e13cf1917b","Type":"ContainerDied","Data":"4608140203d245d18679ad7e6432b15765ed1c10bb2a4a0fc92d01aca22ae25c"}
Oct 29 22:16:46.195405 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.195332    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75" event={"ID":"f85ac047-ae41-4a6d-9711-87e13cf1917b","Type":"ContainerDied","Data":"f8cf057c17adaa37afc48334e1c36d8e22cffda1a2df70e45842bd19f01943d8"}
Oct 29 22:16:46.195405 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.195360    7082 scope.go:122] "RemoveContainer" containerID="4608140203d245d18679ad7e6432b15765ed1c10bb2a4a0fc92d01aca22ae25c"
Oct 29 22:16:46.197119 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.197092    7082 generic.go:356] "Generic (PLEG): container finished" podID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerID="bccbd6fa084f2a6ee5713128fc2175c95a417810cbcba8583c24b8cce74c358e" exitCode=0
Oct 29 22:16:46.197282 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.197180    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" event={"ID":"65dfc02a-c81d-4366-a1a5-a5808d335aaf","Type":"ContainerDied","Data":"bccbd6fa084f2a6ee5713128fc2175c95a417810cbcba8583c24b8cce74c358e"}
Oct 29 22:16:46.197282 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.197197    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"
Oct 29 22:16:46.197282 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.197243    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4" event={"ID":"65dfc02a-c81d-4366-a1a5-a5808d335aaf","Type":"ContainerDied","Data":"60882761c6bc8cef65b178f934791ba1389e544f21080fa60996b6fdca4aa977"}
Oct 29 22:16:46.199565 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.199471    7082 generic.go:356] "Generic (PLEG): container finished" podID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerID="2faa7e03ec44d32988b2b89c3b88e65a35e3d02a1926e28262215b309747cb4b" exitCode=0
Oct 29 22:16:46.199701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.199565    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" event={"ID":"9c6e140c-ea25-446f-a3b2-42d49e6634e5","Type":"ContainerDied","Data":"2faa7e03ec44d32988b2b89c3b88e65a35e3d02a1926e28262215b309747cb4b"}
Oct 29 22:16:46.199701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.199603    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd" event={"ID":"9c6e140c-ea25-446f-a3b2-42d49e6634e5","Type":"ContainerDied","Data":"f8b0ffd3297ce9fa7332d1ebc5b587d769a3b2eee31b5f5e571566ca3bc6e2ff"}
Oct 29 22:16:46.199940 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.199918    7082 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"
Oct 29 22:16:46.208277 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.208261    7082 scope.go:122] "RemoveContainer" containerID="4608140203d245d18679ad7e6432b15765ed1c10bb2a4a0fc92d01aca22ae25c"
Oct 29 22:16:46.208578 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:16:46.208557    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4608140203d245d18679ad7e6432b15765ed1c10bb2a4a0fc92d01aca22ae25c\": container with ID starting with 4608140203d245d18679ad7e6432b15765ed1c10bb2a4a0fc92d01aca22ae25c not found: ID does not exist" containerID="4608140203d245d18679ad7e6432b15765ed1c10bb2a4a0fc92d01aca22ae25c"
Oct 29 22:16:46.208710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.208584    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4608140203d245d18679ad7e6432b15765ed1c10bb2a4a0fc92d01aca22ae25c"} err="failed to get container status \"4608140203d245d18679ad7e6432b15765ed1c10bb2a4a0fc92d01aca22ae25c\": rpc error: code = NotFound desc = could not find container \"4608140203d245d18679ad7e6432b15765ed1c10bb2a4a0fc92d01aca22ae25c\": container with ID starting with 4608140203d245d18679ad7e6432b15765ed1c10bb2a4a0fc92d01aca22ae25c not found: ID does not exist"
Oct 29 22:16:46.208710 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.208601    7082 scope.go:122] "RemoveContainer" containerID="bccbd6fa084f2a6ee5713128fc2175c95a417810cbcba8583c24b8cce74c358e"
Oct 29 22:16:46.219560 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.219529    7082 scope.go:122] "RemoveContainer" containerID="bccbd6fa084f2a6ee5713128fc2175c95a417810cbcba8583c24b8cce74c358e"
Oct 29 22:16:46.219926 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:16:46.219899    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bccbd6fa084f2a6ee5713128fc2175c95a417810cbcba8583c24b8cce74c358e\": container with ID starting with bccbd6fa084f2a6ee5713128fc2175c95a417810cbcba8583c24b8cce74c358e not found: ID does not exist" containerID="bccbd6fa084f2a6ee5713128fc2175c95a417810cbcba8583c24b8cce74c358e"
Oct 29 22:16:46.220022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.219937    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bccbd6fa084f2a6ee5713128fc2175c95a417810cbcba8583c24b8cce74c358e"} err="failed to get container status \"bccbd6fa084f2a6ee5713128fc2175c95a417810cbcba8583c24b8cce74c358e\": rpc error: code = NotFound desc = could not find container \"bccbd6fa084f2a6ee5713128fc2175c95a417810cbcba8583c24b8cce74c358e\": container with ID starting with bccbd6fa084f2a6ee5713128fc2175c95a417810cbcba8583c24b8cce74c358e not found: ID does not exist"
Oct 29 22:16:46.220022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.219961    7082 scope.go:122] "RemoveContainer" containerID="2faa7e03ec44d32988b2b89c3b88e65a35e3d02a1926e28262215b309747cb4b"
Oct 29 22:16:46.231051 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.231035    7082 scope.go:122] "RemoveContainer" containerID="6810b68ab6cc5c4e5422e7350daec0aa56237daac3fc7651529ddab5dd2f7817"
Oct 29 22:16:46.247510 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.247206    7082 scope.go:122] "RemoveContainer" containerID="2faa7e03ec44d32988b2b89c3b88e65a35e3d02a1926e28262215b309747cb4b"
Oct 29 22:16:46.247652 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:16:46.247612    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2faa7e03ec44d32988b2b89c3b88e65a35e3d02a1926e28262215b309747cb4b\": container with ID starting with 2faa7e03ec44d32988b2b89c3b88e65a35e3d02a1926e28262215b309747cb4b not found: ID does not exist" containerID="2faa7e03ec44d32988b2b89c3b88e65a35e3d02a1926e28262215b309747cb4b"
Oct 29 22:16:46.247731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.247662    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2faa7e03ec44d32988b2b89c3b88e65a35e3d02a1926e28262215b309747cb4b"} err="failed to get container status \"2faa7e03ec44d32988b2b89c3b88e65a35e3d02a1926e28262215b309747cb4b\": rpc error: code = NotFound desc = could not find container \"2faa7e03ec44d32988b2b89c3b88e65a35e3d02a1926e28262215b309747cb4b\": container with ID starting with 2faa7e03ec44d32988b2b89c3b88e65a35e3d02a1926e28262215b309747cb4b not found: ID does not exist"
Oct 29 22:16:46.247731 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.247683    7082 scope.go:122] "RemoveContainer" containerID="6810b68ab6cc5c4e5422e7350daec0aa56237daac3fc7651529ddab5dd2f7817"
Oct 29 22:16:46.248063 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:16:46.248026    7082 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6810b68ab6cc5c4e5422e7350daec0aa56237daac3fc7651529ddab5dd2f7817\": container with ID starting with 6810b68ab6cc5c4e5422e7350daec0aa56237daac3fc7651529ddab5dd2f7817 not found: ID does not exist" containerID="6810b68ab6cc5c4e5422e7350daec0aa56237daac3fc7651529ddab5dd2f7817"
Oct 29 22:16:46.248161 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.248059    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"]
Oct 29 22:16:46.248161 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.248071    7082 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6810b68ab6cc5c4e5422e7350daec0aa56237daac3fc7651529ddab5dd2f7817"} err="failed to get container status \"6810b68ab6cc5c4e5422e7350daec0aa56237daac3fc7651529ddab5dd2f7817\": rpc error: code = NotFound desc = could not find container \"6810b68ab6cc5c4e5422e7350daec0aa56237daac3fc7651529ddab5dd2f7817\": container with ID starting with 6810b68ab6cc5c4e5422e7350daec0aa56237daac3fc7651529ddab5dd2f7817 not found: ID does not exist"
Oct 29 22:16:46.258305 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.258283    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-hpmr4"]
Oct 29 22:16:46.286904 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.286860    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"]
Oct 29 22:16:46.290597 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.290568    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-llz75"]
Oct 29 22:16:46.322958 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.322917    7082 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"]
Oct 29 22:16:46.324876 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.324854    7082 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-kzcrd"]
Oct 29 22:16:46.763254 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:46.763216    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8468f79957-dxp8f_fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c/authorino/0.log"
Oct 29 22:16:47.630411 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:47.630377    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" path="/var/lib/kubelet/pods/65dfc02a-c81d-4366-a1a5-a5808d335aaf/volumes"
Oct 29 22:16:47.630848 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:47.630834    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" path="/var/lib/kubelet/pods/9c6e140c-ea25-446f-a3b2-42d49e6634e5/volumes"
Oct 29 22:16:47.631253 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:47.631228    7082 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" path="/var/lib/kubelet/pods/f85ac047-ae41-4a6d-9711-87e13cf1917b/volumes"
Oct 29 22:16:57.647523 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:57.647490    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9cw8s_522ce2f8-9d09-478b-8575-b2dabf74a61e/manager/0.log"
Oct 29 22:16:58.412339 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:16:58.412302    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:16:59.824928 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:16:59.824896    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0eecee06-ztv4z_7fa68641-c5bc-46e1-8492-f46183338fe7/revoke-keys/0.log"
Oct 29 22:17:00.006149 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:00.006120    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-28c639c5-x5w72_433477a8-5832-4475-989c-e01be3c42e88/revoke-keys/0.log"
Oct 29 22:17:00.195701 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:00.195597    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3eaa59b5-tzt5v_2351c336-be09-42b5-a7d9-7b1806e4eccc/revoke-keys/0.log"
Oct 29 22:17:00.388179 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:00.388152    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3eedc34b-fzflz_03cd98c0-0f31-4b7d-8c0d-c15e827315c8/revoke-keys/0.log"
Oct 29 22:17:00.584329 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:00.584300    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4ae5fb64-5vnxs_b720e32b-c20f-48a1-ac2a-b26405ef141f/revoke-keys/0.log"
Oct 29 22:17:00.773074 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:00.773038    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-75d26cc0-prwlq_4646467d-1071-4a13-908c-42cadafb711c/revoke-keys/0.log"
Oct 29 22:17:00.975123 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:00.975049    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-767ac17e-rlcls_86a8d55f-8e42-4f30-8f53-0fa9af1b2cde/revoke-keys/0.log"
Oct 29 22:17:01.168601 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:01.168567    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-819d7332-65s7r_d9b4519a-2ca2-4da5-9b8b-44b3a8b4cf29/revoke-keys/0.log"
Oct 29 22:17:01.361988 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:01.361958    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-98ad10af-94rzf_59f43c8d-9dac-4f7d-94e1-e0100cd449d0/revoke-keys/0.log"
Oct 29 22:17:03.460483 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:03.460450    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8468f79957-dxp8f_fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c/authorino/0.log"
Oct 29 22:17:03.944954 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:03.944925    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_developer-portal-controller-66d8dbb8c8-b2ll6_ba1d0015-3c5e-450d-a435-d33ab885f265/manager/0.log"
Oct 29 22:17:04.936382 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:04.936345    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6b575997cc-zthg8_b8178c0e-7643-499f-9dc0-e04329809348/manager/1.log"
Oct 29 22:17:05.148751 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:05.148714    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-f7d6b5c7c-7sl7d_72d022e0-2c45-448f-8bfb-77d5d75d6f8f/limitador/0.log"
Oct 29 22:17:05.984100 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:05.984055    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6b575997cc-zthg8_b8178c0e-7643-499f-9dc0-e04329809348/manager/0.log"
Oct 29 22:17:06.550268 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:06.550227    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-649697ffd5j9s7t_16d41fba-a15d-4b5c-b42f-54457db55a5a/istio-proxy/0.log"
Oct 29 22:17:06.941136 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:06.941056    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-g8rvj_1aee5f5c-e3c5-4d61-9082-417a3fd2795d/discovery/0.log"
Oct 29 22:17:07.089181 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:07.089149    7082 scope.go:122] "RemoveContainer" containerID="1852ba3561573d24f4e0fe909d430ab0391d634976d761dac89c590011e86eca"
Oct 29 22:17:07.099295 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:07.099274    7082 scope.go:122] "RemoveContainer" containerID="17b43d19110d84be9fb43d8dd1a0a4ec04e47a2194bdc7e37b0857d88fb2d45b"
Oct 29 22:17:07.108656 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:07.108617    7082 scope.go:122] "RemoveContainer" containerID="8ebdd95e16ec7b6bef435eb4cde4d2a0cf8a1920956fbe8d9872efe7dbe53310"
Oct 29 22:17:07.117840 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:07.117818    7082 scope.go:122] "RemoveContainer" containerID="e4976ce9a8dddb2824354d814de6643a8a2b7e042eef84935ae100ebcae96168"
Oct 29 22:17:07.571759 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:07.571729    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64545848b5-ffqsz_4e810e5c-e197-4b47-b906-95e8b88f5586/kube-auth-proxy/0.log"
Oct 29 22:17:07.948938 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:07.948867    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-9xmcs_508e1b3c-a7fe-4c6f-bc38-128cec822dc6/istio-proxy/0.log"
Oct 29 22:17:08.705994 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:08.705942    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64dd656c99-q6wwv_1b93e523-6093-4484-98c4-ee9762845a2c/istio-proxy/0.log"
Oct 29 22:17:10.064056 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:10.064003    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-vzwvt_f069baec-59d9-4c9e-b5a2-c25261d01cf4/main/0.log"
Oct 29 22:17:10.642217 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:10.642183    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-tbk7p_0bc3bfd2-1b15-4edd-ad1f-de9cdf0d7496/main/0.log"
Oct 29 22:17:11.028055 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:11.027962    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-zqhtx_4684c6fd-8ac7-4005-a20d-48396c0ab981/main/0.log"
Oct 29 22:17:11.222792 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:11.222765    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-kd2xs_c32b079b-eba6-4e6f-8f79-f9c3cb86de99/main/0.log"
Oct 29 22:17:11.413177 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:11.413145    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-4fm7q_2eb1b9db-8f2e-448e-8b42-96025377751b/main/0.log"
Oct 29 22:17:27.785803 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:27.785721    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vth7d_ae94d17c-b881-46e1-8063-e88c9dc53369/global-pull-secret-syncer/0.log"
Oct 29 22:17:28.410309 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:17:28.410278    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:17:28.647134 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:28.647109    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vwk66_324d72a4-a641-4b09-99d6-d682b5e2c958/konnectivity-agent/0.log"
Oct 29 22:17:29.018242 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:29.018220    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-38.ec2.internal_39cf47f8a2a071911c6009d122ac3420/haproxy/0.log"
Oct 29 22:17:35.472018 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:35.471980    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8468f79957-dxp8f_fc83c5e9-bbc9-4365-bbc0-1068b5c44c8c/authorino/0.log"
Oct 29 22:17:35.775792 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:35.775721    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_developer-portal-controller-66d8dbb8c8-b2ll6_ba1d0015-3c5e-450d-a435-d33ab885f265/manager/0.log"
Oct 29 22:17:36.127527 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:36.127489    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6b575997cc-zthg8_b8178c0e-7643-499f-9dc0-e04329809348/manager/0.log"
Oct 29 22:17:36.521052 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:36.520961    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6b575997cc-zthg8_b8178c0e-7643-499f-9dc0-e04329809348/manager/1.log"
Oct 29 22:17:37.903864 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:37.903829    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-f7d6b5c7c-7sl7d_72d022e0-2c45-448f-8bfb-77d5d75d6f8f/limitador/0.log"
Oct 29 22:17:39.010750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.010712    7082 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"]
Oct 29 22:17:39.011195 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011181    7082 cpu_manager.go:414] "containerMap: removing container" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main"
Oct 29 22:17:39.011254 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011197    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main"
Oct 29 22:17:39.011254 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011214    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerName="istio-proxy"
Oct 29 22:17:39.011254 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011220    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerName="istio-proxy"
Oct 29 22:17:39.011254 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011229    7082 cpu_manager.go:414] "containerMap: removing container" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy"
Oct 29 22:17:39.011254 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011235    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy"
Oct 29 22:17:39.011254 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011253    7082 cpu_manager.go:414] "containerMap: removing container" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy"
Oct 29 22:17:39.011529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011261    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy"
Oct 29 22:17:39.011529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011296    7082 cpu_manager.go:414] "containerMap: removing container" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerName="istio-proxy"
Oct 29 22:17:39.011529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011304    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerName="istio-proxy"
Oct 29 22:17:39.011529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011318    7082 cpu_manager.go:414] "containerMap: removing container" podUID="433477a8-5832-4475-989c-e01be3c42e88" containerName="revoke-keys"
Oct 29 22:17:39.011529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011323    7082 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433477a8-5832-4475-989c-e01be3c42e88" containerName="revoke-keys"
Oct 29 22:17:39.011529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011442    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c6e140c-ea25-446f-a3b2-42d49e6634e5" containerName="istio-proxy"
Oct 29 22:17:39.011529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011459    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="433477a8-5832-4475-989c-e01be3c42e88" containerName="revoke-keys"
Oct 29 22:17:39.011529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011470    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="55aac7b4-d385-4ba9-b5a0-7528b481f142" containerName="main"
Oct 29 22:17:39.011529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011481    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="65dfc02a-c81d-4366-a1a5-a5808d335aaf" containerName="istio-proxy"
Oct 29 22:17:39.011529 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.011490    7082 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85ac047-ae41-4a6d-9711-87e13cf1917b" containerName="istio-proxy"
Oct 29 22:17:39.015022 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.015003    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.030985 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.030958    7082 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bzkjc\"/\"default-dockercfg-78mjz\""
Oct 29 22:17:39.033926 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.033906    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bzkjc\"/\"openshift-service-ca.crt\""
Oct 29 22:17:39.042703 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.042680    7082 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bzkjc\"/\"kube-root-ca.crt\""
Oct 29 22:17:39.045074 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.045055    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"]
Oct 29 22:17:39.123255 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.123224    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-sys\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.123454 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.123275    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgzr4\" (UniqueName: \"kubernetes.io/projected/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-kube-api-access-dgzr4\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.123454 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.123425    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-podres\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.123593 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.123489    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-proc\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.123593 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.123560    7082 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-lib-modules\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.225006 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.224968    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-proc\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.225210 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.225020    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-lib-modules\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.225210 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.225065    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-sys\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.225210 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.225101    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-proc\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.225210 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.225116    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzr4\" (UniqueName: \"kubernetes.io/projected/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-kube-api-access-dgzr4\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.225210 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.225187    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-sys\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.225210 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.225191    7082 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-podres\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.225516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.225237    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-podres\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.225516 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.225270    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-lib-modules\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.248307 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.248270    7082 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgzr4\" (UniqueName: \"kubernetes.io/projected/5d25bdd9-8166-4a6d-824f-cdbcf05f2500-kube-api-access-dgzr4\") pod \"perf-node-gather-daemonset-lhh66\" (UID: \"5d25bdd9-8166-4a6d-824f-cdbcf05f2500\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.327861 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.327827    7082 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:39.632667 ip-10-0-136-38 kubenswrapper[7082]: W1029 22:17:39.632612    7082 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5d25bdd9_8166_4a6d_824f_cdbcf05f2500.slice/crio-1d523257328d8a4654370740b662226763c465e658b4599023f659e6f1eba004 WatchSource:0}: Error finding container 1d523257328d8a4654370740b662226763c465e658b4599023f659e6f1eba004: Status 404 returned error can't find the container with id 1d523257328d8a4654370740b662226763c465e658b4599023f659e6f1eba004
Oct 29 22:17:39.634477 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.634451    7082 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"]
Oct 29 22:17:39.779543 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.779513    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7314f42f-afe2-4c74-8281-760de1e84ba6/alertmanager/0.log"
Oct 29 22:17:39.908800 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:39.908724    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7314f42f-afe2-4c74-8281-760de1e84ba6/config-reloader/0.log"
Oct 29 22:17:40.027440 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:40.027410    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7314f42f-afe2-4c74-8281-760de1e84ba6/kube-rbac-proxy-web/0.log"
Oct 29 22:17:40.148048 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:40.148001    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7314f42f-afe2-4c74-8281-760de1e84ba6/kube-rbac-proxy/0.log"
Oct 29 22:17:40.269583 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:40.269514    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7314f42f-afe2-4c74-8281-760de1e84ba6/kube-rbac-proxy-metric/0.log"
Oct 29 22:17:40.391256 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:40.391227    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7314f42f-afe2-4c74-8281-760de1e84ba6/prom-label-proxy/0.log"
Oct 29 22:17:40.508636 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:40.508587    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9cw8s_522ce2f8-9d09-478b-8575-b2dabf74a61e/manager/0.log"
Oct 29 22:17:40.509442 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:40.509424    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7314f42f-afe2-4c74-8281-760de1e84ba6/init-config-reloader/0.log"
Oct 29 22:17:40.558225 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:40.558201    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66" event={"ID":"5d25bdd9-8166-4a6d-824f-cdbcf05f2500","Type":"ContainerStarted","Data":"296df20e0252d731529fb6ff6f2f33a9dac64b2f5dccf5382f7bf2cd7e1d5fdb"}
Oct 29 22:17:40.558365 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:40.558243    7082 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66" event={"ID":"5d25bdd9-8166-4a6d-824f-cdbcf05f2500","Type":"ContainerStarted","Data":"1d523257328d8a4654370740b662226763c465e658b4599023f659e6f1eba004"}
Oct 29 22:17:40.558365 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:40.558324    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:40.598899 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:40.598859    7082 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66" podStartSLOduration=2.598845426 podStartE2EDuration="2.598845426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:40.59671688 +0000 UTC m=+1833.096759180" watchObservedRunningTime="2026-10-29 22:17:40.598845426 +0000 UTC m=+1833.098887725"
Oct 29 22:17:40.676750 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:40.676725    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9cw8s_522ce2f8-9d09-478b-8575-b2dabf74a61e/manager/0.log"
Oct 29 22:17:40.766057 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:40.766028    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-xw2d4_066041b1-7eba-4371-9470-101a7bf8fcfc/kube-state-metrics/0.log"
Oct 29 22:17:40.883451 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:40.883365    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-xw2d4_066041b1-7eba-4371-9470-101a7bf8fcfc/kube-rbac-proxy-main/0.log"
Oct 29 22:17:41.004562 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:41.004531    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-xw2d4_066041b1-7eba-4371-9470-101a7bf8fcfc/kube-rbac-proxy-self/0.log"
Oct 29 22:17:41.131635 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:41.131602    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f7fc46b4-vvxlv_baaab207-5534-4ef9-ac3a-9466ea5b900e/metrics-server/0.log"
Oct 29 22:17:41.253815 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:41.253743    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-wc4l2_4c2dd590-c639-42fe-8a02-4945b6403b3b/monitoring-plugin/0.log"
Oct 29 22:17:42.138774 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:42.138742    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wtbq7_3b64908a-fffb-4ec2-8697-51ea5db37f3b/node-exporter/0.log"
Oct 29 22:17:42.254842 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:42.254812    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wtbq7_3b64908a-fffb-4ec2-8697-51ea5db37f3b/kube-rbac-proxy/0.log"
Oct 29 22:17:42.372777 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:42.372748    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wtbq7_3b64908a-fffb-4ec2-8697-51ea5db37f3b/init-textfile/0.log"
Oct 29 22:17:42.498886 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:42.498813    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-js76b_8c893b43-cd05-45a0-9f7b-a2b5d72809c7/kube-rbac-proxy-main/0.log"
Oct 29 22:17:42.616490 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:42.616461    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-js76b_8c893b43-cd05-45a0-9f7b-a2b5d72809c7/kube-rbac-proxy-self/0.log"
Oct 29 22:17:42.737113 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:42.737076    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-js76b_8c893b43-cd05-45a0-9f7b-a2b5d72809c7/openshift-state-metrics/0.log"
Oct 29 22:17:43.425755 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:43.425726    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9cw8s_522ce2f8-9d09-478b-8575-b2dabf74a61e/manager/0.log"
Oct 29 22:17:43.605324 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:43.605300    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9cw8s_522ce2f8-9d09-478b-8575-b2dabf74a61e/manager/0.log"
Oct 29 22:17:44.054492 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:44.054460    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54969c98b4-7zdlx_dab249e6-5406-4ca6-87d5-d6fe41de7754/telemeter-client/0.log"
Oct 29 22:17:44.172992 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:44.172959    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54969c98b4-7zdlx_dab249e6-5406-4ca6-87d5-d6fe41de7754/reload/0.log"
Oct 29 22:17:44.293679 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:44.293657    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54969c98b4-7zdlx_dab249e6-5406-4ca6-87d5-d6fe41de7754/kube-rbac-proxy/0.log"
Oct 29 22:17:44.424648 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:44.424547    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f96f5cd85-t7ndb_b96d72b2-bc45-4962-b529-f08155ad0215/thanos-query/0.log"
Oct 29 22:17:44.545508 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:44.545476    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f96f5cd85-t7ndb_b96d72b2-bc45-4962-b529-f08155ad0215/kube-rbac-proxy-web/0.log"
Oct 29 22:17:44.666469 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:44.666438    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f96f5cd85-t7ndb_b96d72b2-bc45-4962-b529-f08155ad0215/kube-rbac-proxy/0.log"
Oct 29 22:17:44.787573 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:44.787498    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f96f5cd85-t7ndb_b96d72b2-bc45-4962-b529-f08155ad0215/prom-label-proxy/0.log"
Oct 29 22:17:44.910137 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:44.910113    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f96f5cd85-t7ndb_b96d72b2-bc45-4962-b529-f08155ad0215/kube-rbac-proxy-rules/0.log"
Oct 29 22:17:45.029405 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:45.029368    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f96f5cd85-t7ndb_b96d72b2-bc45-4962-b529-f08155ad0215/kube-rbac-proxy-metrics/0.log"
Oct 29 22:17:46.413508 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:46.413456    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9cw8s_522ce2f8-9d09-478b-8575-b2dabf74a61e/manager/0.log"
Oct 29 22:17:46.576853 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:46.576825    7082 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-lhh66"
Oct 29 22:17:46.592808 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:46.592779    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9cw8s_522ce2f8-9d09-478b-8575-b2dabf74a61e/manager/0.log"
Oct 29 22:17:49.651372 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:49.651337    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bd87bd5df-9rfzx_4a9db242-c903-49e3-92a6-fdf982c8f25e/console/0.log"
Oct 29 22:17:50.056841 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:50.056812    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-649697ffd5j9s7t_16d41fba-a15d-4b5c-b42f-54457db55a5a/istio-proxy/0.log"
Oct 29 22:17:50.256292 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:50.256264    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-649697ffd5j9s7t_16d41fba-a15d-4b5c-b42f-54457db55a5a/istio-proxy/0.log"
Oct 29 22:17:54.132035 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:54.132004    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-245p6_3accc688-8a04-4462-9593-228a18548478/dns/0.log"
Oct 29 22:17:54.251136 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:54.251111    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-245p6_3accc688-8a04-4462-9593-228a18548478/kube-rbac-proxy/0.log"
Oct 29 22:17:55.023007 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:55.022975    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8jkmb_a09b130d-4249-4539-add0-e0aba69b752a/dns-node-resolver/0.log"
Oct 29 22:17:55.169092 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:55.169063    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-649697ffd5j9s7t_16d41fba-a15d-4b5c-b42f-54457db55a5a/istio-proxy/0.log"
Oct 29 22:17:56.636499 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:56.636467    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dmgg8_1f41e102-fab5-4206-9502-fbf09515069a/node-ca/0.log"
Oct 29 22:17:58.415616 ip-10-0-136-38 kubenswrapper[7082]: E1029 22:17:58.415576    7082 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2867l" containerName="extractor"
Oct 29 22:17:58.728116 ip-10-0-136-38 kubenswrapper[7082]: I1029 22:17:58.728019    7082 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-g8rvj_1aee5f5c-e3c5-4d61-9082-417a3fd2795d/discovery/0.log"